[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:35 dist-test-slave-c3s8 krb5kdc[21960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:35 dist-test-slave-c3s8 krb5kdc[21960](info): set up 2 sockets
Sep 28 23:43:35 dist-test-slave-c3s8 krb5kdc[21960](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 23:43:36.059559 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:43:36.202888 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34395
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:34395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:36.328137 21973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:36.328456 21973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:36.328500 21973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:36.332975 21973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:36.333055 21973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:36.333079 21973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:36.333097 21973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:36.333113 21973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:36.339188 21973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:34395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:36.340395 21973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:36.341514 21973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:36.350690 21978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:36.351068 21973 server_base.cc:1034] running on GCE node
W20260928 23:43:36.350873 21981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:36.350878 21979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:36.351845 21973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:36.353022 21973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:36.354257 21973 hybrid_clock.cc:648] HybridClock initialized: now 1790639016354226 us; error 43 us; skew 500 ppm
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:36.357601 21973 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:36.358740 21973 webserver.cc:460] Webserver started at http://127.21.112.126:40431/ using document root <none> and password file <none>
I20260928 23:43:36.359377 21973 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:36.359444 21973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:36.359781 21973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:36.362176 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "753088e49332cf0ec756dafad0b751e9"
server_key_iv: "b66f71497b3ccf5d1c1f49c9f9fa17d7"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.362828 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "753088e49332cf0ec756dafad0b751e9"
server_key_iv: "b66f71497b3ccf5d1c1f49c9f9fa17d7"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.366973 21973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:43:36.369887 21988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.371320 21973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:36.371488 21973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "753088e49332cf0ec756dafad0b751e9"
server_key_iv: "b66f71497b3ccf5d1c1f49c9f9fa17d7"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.371619 21973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:36.394524 21973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:36.398783 21973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:36.399044 21973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:36.406695 21973 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:34395
I20260928 23:43:36.406720 22040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:34395 every 8 connection(s)
I20260928 23:43:36.407701 21973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:36.410821 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:36.411041 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 21973
I20260928 23:43:36.411188 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:36.412838 21953 external_mini_cluster.cc:1428] Setting key 5f1aa2ceb918e524ed7cf0d0fa9d7bc3
I20260928 23:43:36.417912 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d: Bootstrap starting.
I20260928 23:43:36.420918 22041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:36.421928 22041 log.cc:824] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:36.424605 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d: No bootstrap required, opened a new log
I20260928 23:43:36.429255 22041 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34395 } }
I20260928 23:43:36.429596 22041 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:36.429654 22041 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae26c6ac651a4b43a2e3edc4d544fb0d, State: Initialized, Role: FOLLOWER
I20260928 23:43:36.430172 22041 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34395 } }
I20260928 23:43:36.430303 22041 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:36.430377 22041 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:36.430478 22041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:36.431802 22041 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34395 } }
I20260928 23:43:36.432245 22041 leader_election.cc:302] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae26c6ac651a4b43a2e3edc4d544fb0d; no voters:
I20260928 23:43:36.432672 22041 leader_election.cc:288] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:36.432855 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:36.433103 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [term 1 LEADER]: Becoming Leader. State: Replica: ae26c6ac651a4b43a2e3edc4d544fb0d, State: Running, Role: LEADER
I20260928 23:43:36.433452 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34395 } }
I20260928 23:43:36.434365 22041 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:36.435631 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34395 } } }
I20260928 23:43:36.435778 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:36.436236 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae26c6ac651a4b43a2e3edc4d544fb0d. Latest consensus state: current_term: 1 leader_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae26c6ac651a4b43a2e3edc4d544fb0d" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34395 } } }
I20260928 23:43:36.436436 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:36.436751 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:36.440351 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:36.443899 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:36.416286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.21.112.126:34395)
0928 23:43:36.416844 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:36.416864 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:36.418086 (+ 1222us) server_negotiation.cc:407] Connection header received
0928 23:43:36.422542 (+ 4456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:36.422568 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:36.423006 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:36.423392 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:36.426714 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.427922 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.429779 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.430361 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.430595 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:36.435940 (+ 5345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:36.435949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:36.435966 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:36.436006 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:36.438817 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:36.439743 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:36.439752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:36.439759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:36.439941 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:36.440497 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:36.440500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:36.440502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:36.441085 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:36.441316 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:36.442108 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 23:43:36.442435 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":155,"threads_started":1}
I20260928 23:43:36.446964 22055 catalog_manager.cc:1347] Generated new cluster ID: cd29ac15236f4033aca838fc1250e2d6
I20260928 23:43:36.447044 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:36.483469 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:36.485071 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:36.491298 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae26c6ac651a4b43a2e3edc4d544fb0d: Generated new TSK 0
I20260928 23:43:36.492214 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:43:36.525315 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34395
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:36.641314 22067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:36.641685 22067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:36.641765 22067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:36.645879 22067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:36.645942 22067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:36.646031 22067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:43:36.652020 22067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:36.653182 22067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:36.654309 22067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:36.664547 22073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:36.664561 22072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:36.664595 22075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:36.664616 22067 server_base.cc:1034] running on GCE node
I20260928 23:43:36.665277 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:36.666069 22067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:36.667327 22067 hybrid_clock.cc:648] HybridClock initialized: now 1790639016667310 us; error 31 us; skew 500 ppm
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:36.670719 22067 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:43:36.671749 22067 webserver.cc:460] Webserver started at http://127.21.112.65:45127/ using document root <none> and password file <none>
I20260928 23:43:36.672474 22067 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:36.672596 22067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:36.672829 22067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:36.675041 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "c00fe8845a9944128b4fe32b6417c4b7"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "2e94bf3a41e7dff569f272b94fa9957b"
server_key_iv: "895c7805e7de0d3e1fb988bbaeb2ecbe"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.675678 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "c00fe8845a9944128b4fe32b6417c4b7"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "2e94bf3a41e7dff569f272b94fa9957b"
server_key_iv: "895c7805e7de0d3e1fb988bbaeb2ecbe"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.680001 22067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:43:36.683125 22082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.684572 22067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:43:36.684733 22067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "c00fe8845a9944128b4fe32b6417c4b7"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "2e94bf3a41e7dff569f272b94fa9957b"
server_key_iv: "895c7805e7de0d3e1fb988bbaeb2ecbe"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.684873 22067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:36.710759 22067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:36.714699 22067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:36.715018 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:36.715694 22067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:36.716645 22067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:36.716717 22067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.716806 22067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:36.716858 22067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.727239 22067 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:35797
I20260928 23:43:36.727262 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:35797 every 8 connection(s)
I20260928 23:43:36.728245 22067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:43:36.732996 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22067
I20260928 23:43:36.733150 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:43:36.733435 21953 external_mini_cluster.cc:1428] Setting key 04be95106bcdf5df43d858936583bf51
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:43:36.743947 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:36.730414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:47249 (local address 127.21.112.126:34395)
0928 23:43:36.730568 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:36.730572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:36.731513 (+ 941us) server_negotiation.cc:407] Connection header received
0928 23:43:36.732901 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:36.732904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:36.732967 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:36.733084 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:36.734604 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.735310 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.736635 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.736884 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.737004 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:36.740020 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:36.740024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:36.740026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:36.740052 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:36.741618 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:36.742188 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:36.742191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:36.742193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:36.742277 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:36.742635 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:36.742639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:36.742640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:36.742847 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:36.742980 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:36.743646 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 23:43:36.743818 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:43:36.745287 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:36.730822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34395 (local address 127.21.112.65:47249)
0928 23:43:36.731331 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:36.731389 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:36.732665 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:36.733318 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:36.733326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:36.733810 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:36.734454 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:36.734471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.735454 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.735457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:36.736502 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:36.736513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.737060 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.737064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:36.737245 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:36.737960 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:36.737978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:36.739807 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:36.741748 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:36.741756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:36.741776 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:36.742072 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:36.742374 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:36.742378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:36.742381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:36.742540 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:36.743052 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:36.743064 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:36.743391 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 23:43:36.743576 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 23:43:36.743966 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:43:36.747192 22196 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34395
I20260928 23:43:36.747478 22196 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:36.748093 22196 heartbeater.cc:511] Master 127.21.112.126:34395 requested a full tablet report, sending...
I20260928 23:43:36.749917 22005 ts_manager.cc:194] Registered new tserver with Master: c00fe8845a9944128b4fe32b6417c4b7 (127.21.112.65:35797)
I20260928 23:43:36.751744 22005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:47249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:43:36.765297 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34395
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:36.884284 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:36.884581 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:36.884657 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:36.889075 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:36.889151 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:36.889258 22201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:43:36.895026 22201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:36.896237 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:36.897358 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:36.907469 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:36.907462 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:36.907455 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:36.907804 22201 server_base.cc:1034] running on GCE node
I20260928 23:43:36.908421 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:36.909152 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:36.910364 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790639016910322 us; error 56 us; skew 500 ppm
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:36.913712 22201 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:43:36.914647 22201 webserver.cc:460] Webserver started at http://127.21.112.66:33833/ using document root <none> and password file <none>
I20260928 23:43:36.915216 22201 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:36.915498 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:36.915812 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:36.918248 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "0e3e79ea0547465c9b0a7f993ea026c2"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "8448442ca4a62b43e85f92d3059db787"
server_key_iv: "0b94ebc11f7a76ae9d98fa54046dc844"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.918895 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "0e3e79ea0547465c9b0a7f993ea026c2"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "8448442ca4a62b43e85f92d3059db787"
server_key_iv: "0b94ebc11f7a76ae9d98fa54046dc844"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.923470 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:43:36.926319 22216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.927676 22201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:36.927801 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "0e3e79ea0547465c9b0a7f993ea026c2"
format_stamp: "Formatted at 2026-09-28 23:43:36 on dist-test-slave-c3s8"
server_key: "8448442ca4a62b43e85f92d3059db787"
server_key_iv: "0b94ebc11f7a76ae9d98fa54046dc844"
server_key_version: "encryptionkey@0"
I20260928 23:43:36.927903 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:36.950371 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:36.954042 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:36.954216 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:36.954793 22201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:36.955606 22201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:36.955657 22201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.955698 22201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:36.955713 22201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:36.965866 22201 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:33503
I20260928 23:43:36.965883 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:33503 every 8 connection(s)
I20260928 23:43:36.966878 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:43:36.973006 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22201
I20260928 23:43:36.973143 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:43:36.973407 21953 external_mini_cluster.cc:1428] Setting key ae626e068e8c0169c275b8f92fb79dad
Sep 28 23:43:36 dist-test-slave-c3s8 krb5kdc[21960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639016, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:36.983093 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:36.968940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:58441 (local address 127.21.112.126:34395)
0928 23:43:36.969096 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:36.969101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:36.970134 (+ 1033us) server_negotiation.cc:407] Connection header received
0928 23:43:36.971417 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:36.971420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:36.971471 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:36.971550 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:36.972939 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.973793 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.975025 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.975263 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.975404 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:36.978765 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:36.978769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:36.978771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:36.978799 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:36.980651 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:36.981274 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:36.981277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:36.981280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:36.981369 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:36.981798 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:36.981803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:36.981805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:36.982100 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:36.982212 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:36.982791 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 23:43:36.982923 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:43:36.984388 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:36.969376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34395 (local address 127.21.112.66:58441)
0928 23:43:36.969974 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:36.970022 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:36.971174 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:36.971818 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:36.971825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:36.972212 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:36.972759 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:36.972774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.973956 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.973960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:36.974884 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:36.974895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:36.975450 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:36.975455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:36.975666 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:36.976423 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:36.976441 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:36.978514 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:36.980801 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:36.980808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:36.980835 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:36.981143 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:36.981471 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:36.981475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:36.981477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:36.981697 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:36.982219 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:36.982229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:36.982521 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 23:43:36.982727 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 23:43:36.983133 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":172,"threads_started":1}
I20260928 23:43:36.985852 22330 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34395
I20260928 23:43:36.986106 22330 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:36.986747 22330 heartbeater.cc:511] Master 127.21.112.126:34395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:43:36.988080 22005 ts_manager.cc:194] Registered new tserver with Master: 0e3e79ea0547465c9b0a7f993ea026c2 (127.21.112.66:33503)
I20260928 23:43:36.988768 22005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:58441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:43:37.006561 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34395
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:37.122094 22335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:37.122361 22335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:37.122407 22335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:37.126523 22335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:37.126590 22335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:37.126693 22335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:43:37.132660 22335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:37.133906 22335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:37.135078 22335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:37.145323 22341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:37.145364 22335 server_base.cc:1034] running on GCE node
W20260928 23:43:37.145323 22343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:37.145382 22340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:37.146039 22335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:37.146723 22335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:37.147940 22335 hybrid_clock.cc:648] HybridClock initialized: now 1790639017147912 us; error 65 us; skew 500 ppm
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639017, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:37.150995 22335 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:43:37.152015 22335 webserver.cc:460] Webserver started at http://127.21.112.67:42291/ using document root <none> and password file <none>
I20260928 23:43:37.152643 22335 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:37.152706 22335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:37.152885 22335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:37.155038 22335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "bf92e0e775f4426aa87cd7dae100ec7c"
format_stamp: "Formatted at 2026-09-28 23:43:37 on dist-test-slave-c3s8"
server_key: "3c6f727eb10ecca02d12df4fa12c0627"
server_key_iv: "468f773931d66f37c5f880d805e221cc"
server_key_version: "encryptionkey@0"
I20260928 23:43:37.155580 22335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "bf92e0e775f4426aa87cd7dae100ec7c"
format_stamp: "Formatted at 2026-09-28 23:43:37 on dist-test-slave-c3s8"
server_key: "3c6f727eb10ecca02d12df4fa12c0627"
server_key_iv: "468f773931d66f37c5f880d805e221cc"
server_key_version: "encryptionkey@0"
I20260928 23:43:37.159448 22335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:43:37.162127 22350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:37.163496 22335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:37.163594 22335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "bf92e0e775f4426aa87cd7dae100ec7c"
format_stamp: "Formatted at 2026-09-28 23:43:37 on dist-test-slave-c3s8"
server_key: "3c6f727eb10ecca02d12df4fa12c0627"
server_key_iv: "468f773931d66f37c5f880d805e221cc"
server_key_version: "encryptionkey@0"
I20260928 23:43:37.163792 22335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:37.175622 22335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:37.179226 22335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:37.179442 22335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:37.180075 22335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:37.180922 22335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:37.180965 22335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:37.181005 22335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:37.181061 22335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:37.190275 22335 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:41389
I20260928 23:43:37.190333 22463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:41389 every 8 connection(s)
I20260928 23:43:37.191241 22335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:43:37.193514 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22335
I20260928 23:43:37.193723 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:43:37.194031 21953 external_mini_cluster.cc:1428] Setting key 164558549b24e68a0738f5658b062c0d
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[21960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639017, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:37.206730 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:37.193228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:53243 (local address 127.21.112.126:34395)
0928 23:43:37.193416 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:37.193421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:37.194476 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 23:43:37.196104 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:37.196107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:37.196151 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:37.196222 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:37.197663 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.198187 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.199354 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.199570 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.199654 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:37.202284 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:37.202287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:37.202290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:37.202317 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:37.204339 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:37.205048 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:37.205051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:37.205053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:37.205144 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:37.205507 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:37.205511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:37.205512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:37.205752 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:37.205858 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:37.206406 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 23:43:37.206544 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:43:37.208417 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:37.193708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34395 (local address 127.21.112.67:53243)
0928 23:43:37.194293 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:37.194356 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:37.195885 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:37.196408 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:37.196417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:37.196943 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:37.197468 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:37.197482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.198303 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.198306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:37.199230 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:37.199240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.199706 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.199708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:37.199845 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:37.200417 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:37.200434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:37.202038 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:37.204466 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:37.204474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:37.204521 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:37.204952 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:37.205244 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:37.205247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:37.205249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:37.205404 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:37.205867 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:37.205876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:37.206147 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 23:43:37.206348 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 23:43:37.206692 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":114,"threads_started":1}
I20260928 23:43:37.209946 22464 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34395
I20260928 23:43:37.210232 22464 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:37.210860 22464 heartbeater.cc:511] Master 127.21.112.126:34395 requested a full tablet report, sending...
I20260928 23:43:37.211949 22005 ts_manager.cc:194] Registered new tserver with Master: bf92e0e775f4426aa87cd7dae100ec7c (127.21.112.67:41389)
I20260928 23:43:37.212697 22005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:53243
I20260928 23:43:37.223308 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[21960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:43:37.250941 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:37.241005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.21.112.65:35797)
0928 23:43:37.241282 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:37.241301 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:37.241336 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:43:37.241394 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:37.241399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:37.241671 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:37.241812 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:37.242501 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.243363 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.244259 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.244633 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.244721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:37.247234 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:37.247243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:37.247256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:37.247279 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:37.249013 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:37.249475 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:37.249483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:37.249484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:37.249697 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:37.249990 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:37.249993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:37.249995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:37.250279 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:37.250435 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:37.250652 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 23:43:37.250798 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
W20260928 23:43:37.251978 22110 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60872
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[21960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[21960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:43:37.279397 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:37.270411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.21.112.65:35797)
0928 23:43:37.270636 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:37.270645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:37.270701 (+ 56us) server_negotiation.cc:407] Connection header received
0928 23:43:37.270901 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:37.270904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:37.270948 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:37.271037 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:37.271759 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.272273 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.273135 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:37.273364 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:37.273474 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:37.275960 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:37.275963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:37.275965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:37.275988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:37.277637 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:37.278144 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:37.278147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:37.278149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:37.278245 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:37.278538 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:37.278541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:37.278543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:37.278738 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:37.278870 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:37.279128 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 23:43:37.279266 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:43:37.280757 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22067
I20260928 23:43:37.287253 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22201
I20260928 23:43:37.295250 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22335
I20260928 23:43:37.300853 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 21973
2026-09-28T23:43:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2341 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[22478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[22478](info): set up 2 sockets
Sep 28 23:43:37 dist-test-slave-c3s8 krb5kdc[22478](info): commencing operation
krb5kdc: starting...
W20260928 23:43:38.457901 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 23:43:38.714661 21953 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.257s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 23:43:38.950466 21953 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.236s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 23:43:39.197183 21953 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.247s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/test-user.keytab.
W20260928 23:43:39.946566 21953 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.749s user 0.000s sys 0.002s
Sep 28 23:43:39 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
W20260928 23:43:40.969130 21953 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.112.126: real 0.998s user 0.000s sys 0.003s
I20260928 23:43:40.969825 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34431
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:34431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:41.090876 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:41.091224 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:41.091302 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:41.095669 22491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:41.095777 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:41.095816 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:41.095873 22491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:41.095923 22491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:41.101612 22491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:34431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:41.102864 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:41.103995 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:41.114200 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:41.114701 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:41.114858 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:41.116096 22491 server_base.cc:1034] running on GCE node
I20260928 23:43:41.117044 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:41.118378 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:41.119594 22491 hybrid_clock.cc:648] HybridClock initialized: now 1790639021119569 us; error 40 us; skew 500 ppm
Sep 28 23:43:41 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639021, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:41.127398 22491 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:41.128548 22491 webserver.cc:460] Webserver started at http://127.21.112.126:44639/ using document root <none> and password file <none>
I20260928 23:43:41.129268 22491 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:41.129365 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:41.129664 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:41.131950 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "22326cd2b6e04e30bee6a4689c6a503b"
format_stamp: "Formatted at 2026-09-28 23:43:41 on dist-test-slave-c3s8"
server_key: "55f0f6fe509edbfb763f562f1eb43b6a"
server_key_iv: "43d6146a596c599110ebe8061e5b7fd7"
server_key_version: "encryptionkey@0"
I20260928 23:43:41.132553 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "22326cd2b6e04e30bee6a4689c6a503b"
format_stamp: "Formatted at 2026-09-28 23:43:41 on dist-test-slave-c3s8"
server_key: "55f0f6fe509edbfb763f562f1eb43b6a"
server_key_iv: "43d6146a596c599110ebe8061e5b7fd7"
server_key_version: "encryptionkey@0"
I20260928 23:43:41.136864 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:43:41.139662 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:41.141040 22491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:43:41.141186 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "22326cd2b6e04e30bee6a4689c6a503b"
format_stamp: "Formatted at 2026-09-28 23:43:41 on dist-test-slave-c3s8"
server_key: "55f0f6fe509edbfb763f562f1eb43b6a"
server_key_iv: "43d6146a596c599110ebe8061e5b7fd7"
server_key_version: "encryptionkey@0"
I20260928 23:43:41.141311 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:41.152081 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:41.155675 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:41.155896 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:41.163646 22491 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:34431
I20260928 23:43:41.163647 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:34431 every 8 connection(s)
I20260928 23:43:41.164706 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:41.167064 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22491
I20260928 23:43:41.167172 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:41.167485 21953 external_mini_cluster.cc:1428] Setting key 7fdadcd47ab4f1d15c157c05349e1140
I20260928 23:43:41.168210 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:41.176760 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b: Bootstrap starting.
Sep 28 23:43:41 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:41.179622 22559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:41.180393 22559 log.cc:824] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:41.182557 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b: No bootstrap required, opened a new log
I20260928 23:43:41.183092 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:41.169054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.21.112.126:34431)
0928 23:43:41.169532 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:41.169605 (+ 73us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:41.169645 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:43:41.170441 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:41.170460 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:41.170863 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:41.171247 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:41.172103 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:41.173229 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:41.174255 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:41.174787 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:41.174976 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:41.177973 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:41.177978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:41.177992 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:41.178022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:41.179924 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:41.180399 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:41.180406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:41.180419 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:41.180564 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:41.180877 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:41.180882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:41.180883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:41.181304 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:41.181514 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:41.181963 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 23:43:41.182276 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260928 23:43:41.186647 22559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22326cd2b6e04e30bee6a4689c6a503b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34431 } }
I20260928 23:43:41.186911 22559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:41.186996 22559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22326cd2b6e04e30bee6a4689c6a503b, State: Initialized, Role: FOLLOWER
I20260928 23:43:41.187503 22559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22326cd2b6e04e30bee6a4689c6a503b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34431 } }
I20260928 23:43:41.187661 22559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:41.187748 22559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:41.187856 22559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:41.272042 22559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22326cd2b6e04e30bee6a4689c6a503b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34431 } }
I20260928 23:43:41.272691 22559 leader_election.cc:302] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22326cd2b6e04e30bee6a4689c6a503b; no voters:
I20260928 23:43:41.273178 22559 leader_election.cc:288] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:41.273352 22564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:41.273738 22564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [term 1 LEADER]: Becoming Leader. State: Replica: 22326cd2b6e04e30bee6a4689c6a503b, State: Running, Role: LEADER
I20260928 23:43:41.274222 22564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22326cd2b6e04e30bee6a4689c6a503b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34431 } }
I20260928 23:43:41.274943 22559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:43:41.276291 22565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22326cd2b6e04e30bee6a4689c6a503b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22326cd2b6e04e30bee6a4689c6a503b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34431 } } }
I20260928 23:43:41.276446 22565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:41.276721 22566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22326cd2b6e04e30bee6a4689c6a503b. Latest consensus state: current_term: 1 leader_uuid: "22326cd2b6e04e30bee6a4689c6a503b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22326cd2b6e04e30bee6a4689c6a503b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34431 } } }
I20260928 23:43:41.276893 22566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:41.276888 22573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:41.280323 22573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:41.286960 22573 catalog_manager.cc:1347] Generated new cluster ID: 05ad81280f564036b422f1c69536bfcc
I20260928 23:43:41.287153 22573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:41.300305 22573 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:41.301970 22573 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:41.319461 22573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22326cd2b6e04e30bee6a4689c6a503b: Generated new TSK 0
I20260928 23:43:41.320327 22573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
W20260928 23:43:41.893488 21953 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.112.65: real 0.557s user 0.001s sys 0.003s
I20260928 23:43:41.894105 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34431
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:42.012603 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:42.012933 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:42.013051 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:42.018756 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:42.018865 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:42.018971 22585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:43:42.025683 22585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:42.026883 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:42.027968 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:42.038615 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:42.038628 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:42.038735 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:42.038987 22585 server_base.cc:1034] running on GCE node
I20260928 23:43:42.039484 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:42.040326 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:42.041606 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790639022041584 us; error 39 us; skew 500 ppm
Sep 28 23:43:42 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639022, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:42.045189 22585 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:43:42.046339 22585 webserver.cc:460] Webserver started at http://127.21.112.65:37295/ using document root <none> and password file <none>
I20260928 23:43:42.047027 22585 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:42.047127 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:42.047381 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:42.049657 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "5a75f37089ed44e4ba1d41ea38354f62"
format_stamp: "Formatted at 2026-09-28 23:43:42 on dist-test-slave-c3s8"
server_key: "0142dee930c4f12c6f7737f28947841e"
server_key_iv: "c533bbf0c7fba93e53613c0d4e440d72"
server_key_version: "encryptionkey@0"
I20260928 23:43:42.050319 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "5a75f37089ed44e4ba1d41ea38354f62"
format_stamp: "Formatted at 2026-09-28 23:43:42 on dist-test-slave-c3s8"
server_key: "0142dee930c4f12c6f7737f28947841e"
server_key_iv: "c533bbf0c7fba93e53613c0d4e440d72"
server_key_version: "encryptionkey@0"
I20260928 23:43:42.054641 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:43:42.057379 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:42.058837 22585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:42.059018 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "5a75f37089ed44e4ba1d41ea38354f62"
format_stamp: "Formatted at 2026-09-28 23:43:42 on dist-test-slave-c3s8"
server_key: "0142dee930c4f12c6f7737f28947841e"
server_key_iv: "c533bbf0c7fba93e53613c0d4e440d72"
server_key_version: "encryptionkey@0"
I20260928 23:43:42.059152 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:42.088151 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:42.091886 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:42.092113 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:42.092761 22585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:42.093673 22585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:42.093750 22585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:42.093863 22585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:42.093912 22585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:42.103281 22585 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:40359
I20260928 23:43:42.103332 22713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:40359 every 8 connection(s)
I20260928 23:43:42.104485 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:43:42.111501 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22585
I20260928 23:43:42.111620 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:43:42.111958 21953 external_mini_cluster.cc:1428] Setting key 2b68f4c31aeedb06455d1dd8a36dae34
Sep 28 23:43:42 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639022, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:42.122627 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:42.106537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:37367 (local address 127.21.112.126:34431)
0928 23:43:42.106824 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:42.106828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:42.107763 (+ 935us) server_negotiation.cc:407] Connection header received
0928 23:43:42.109139 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:42.109142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:42.109200 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:42.109288 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:42.110642 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:42.111212 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:42.112906 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:42.113245 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:42.113356 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:42.117624 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:42.117628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:42.117630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:42.117658 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:42.120072 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:42.120674 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:42.120678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:42.120680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:42.120796 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:42.121151 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:42.121154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:42.121157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:42.121478 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:42.121634 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:42.122196 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 23:43:42.122365 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:43:42.123423 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:42.107049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34431 (local address 127.21.112.65:37367)
0928 23:43:42.107593 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:42.107639 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:42.108918 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:42.109483 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:42.109490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:42.109954 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:42.110494 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:42.110509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:42.111393 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:42.111396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:42.112769 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:42.112778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:42.113438 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:42.113443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:42.113758 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:42.115180 (+ 1422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:42.115210 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:42.117323 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:42.120248 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:42.120254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:42.120268 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:42.120552 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:42.120904 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:42.120907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:42.120908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:42.121047 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:42.121637 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:42.121643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:42.121923 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 23:43:42.122095 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 23:43:42.122389 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":129,"threads_started":1}
I20260928 23:43:42.124837 22714 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34431
I20260928 23:43:42.125164 22714 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:42.125803 22714 heartbeater.cc:511] Master 127.21.112.126:34431 requested a full tablet report, sending...
I20260928 23:43:42.127525 22523 ts_manager.cc:194] Registered new tserver with Master: 5a75f37089ed44e4ba1d41ea38354f62 (127.21.112.65:40359)
I20260928 23:43:42.128784 22523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:37367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
W20260928 23:43:42.992575 21953 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.112.66: real 0.879s user 0.002s sys 0.001s
I20260928 23:43:42.993932 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34431
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:43.114830 22720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:43.115110 22720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:43.115156 22720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:43.119344 22720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:43.119407 22720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:43.119468 22720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:43:43.125133 22720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:43.126286 22720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:43.127560 22720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:43:43.131552 22714 heartbeater.cc:503] Master 127.21.112.126:34431 was elected leader, sending a full tablet report...
W20260928 23:43:43.138752 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:43.138752 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:43.139415 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:43.139415 22720 server_base.cc:1034] running on GCE node
I20260928 23:43:43.139955 22720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:43.140743 22720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:43.141944 22720 hybrid_clock.cc:648] HybridClock initialized: now 1790639023141912 us; error 53 us; skew 500 ppm
Sep 28 23:43:43 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639023, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:43.145146 22720 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:43:43.146265 22720 webserver.cc:460] Webserver started at http://127.21.112.66:34113/ using document root <none> and password file <none>
I20260928 23:43:43.146854 22720 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:43.146920 22720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:43.147102 22720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:43.149300 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "36925091951c4f639df1218bc52f102d"
format_stamp: "Formatted at 2026-09-28 23:43:43 on dist-test-slave-c3s8"
server_key: "6c971970987cad30d5cf3d7e479b1a11"
server_key_iv: "b6976927c9d867e72390375412c8ae4e"
server_key_version: "encryptionkey@0"
I20260928 23:43:43.149950 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "36925091951c4f639df1218bc52f102d"
format_stamp: "Formatted at 2026-09-28 23:43:43 on dist-test-slave-c3s8"
server_key: "6c971970987cad30d5cf3d7e479b1a11"
server_key_iv: "b6976927c9d867e72390375412c8ae4e"
server_key_version: "encryptionkey@0"
I20260928 23:43:43.154300 22720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:43:43.157133 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:43.158748 22720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:43.158917 22720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "36925091951c4f639df1218bc52f102d"
format_stamp: "Formatted at 2026-09-28 23:43:43 on dist-test-slave-c3s8"
server_key: "6c971970987cad30d5cf3d7e479b1a11"
server_key_iv: "b6976927c9d867e72390375412c8ae4e"
server_key_version: "encryptionkey@0"
I20260928 23:43:43.159056 22720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:43.194267 22720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:43.198479 22720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:43.198755 22720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:43.199445 22720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:43.200985 22720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:43.201063 22720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:43.201174 22720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:43.201213 22720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:43.211319 22720 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:36071
I20260928 23:43:43.211340 22848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:36071 every 8 connection(s)
I20260928 23:43:43.212329 22720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:43:43.221495 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22720
I20260928 23:43:43.221985 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:43:43.222299 21953 external_mini_cluster.cc:1428] Setting key 46bd335ab256871affe517546db1303b
Sep 28 23:43:43 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639023, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:43.229001 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.214337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:36495 (local address 127.21.112.126:34431)
0928 23:43:43.214645 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.214650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.215392 (+ 742us) server_negotiation.cc:407] Connection header received
0928 23:43:43.216965 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.216969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.217040 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.217182 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:43.218708 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.219348 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.220680 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.220992 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.221119 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.224003 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:43.224007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:43.224009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:43.224036 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:43.225839 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.226518 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.226522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.226525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.226634 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.227056 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.227059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.227061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.227888 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:43.228014 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.228621 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.228797 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
I20260928 23:43:43.230028 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.214715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34431 (local address 127.21.112.66:36495)
0928 23:43:43.215228 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.215276 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.216716 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.217390 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.217397 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.217920 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:43.218545 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.218560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.219497 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.219506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.220542 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.220552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.221170 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.221175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.221374 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.222066 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:43.222112 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:43.223805 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:43.225978 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:43.225985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:43.226006 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:43.226398 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:43.226761 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:43.226765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:43.226767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:43.226939 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:43.228030 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:43.228040 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:43.228348 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.228578 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.228966 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260928 23:43:43.231585 22849 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34431
I20260928 23:43:43.231880 22849 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:43:43.232475 22849 heartbeater.cc:511] Master 127.21.112.126:34431 requested a full tablet report, sending...
I20260928 23:43:43.233794 22523 ts_manager.cc:194] Registered new tserver with Master: 36925091951c4f639df1218bc52f102d (127.21.112.66:36071)
I20260928 23:43:43.234382 22523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:36495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
W20260928 23:43:43.613641 21953 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.112.67: real 0.389s user 0.000s sys 0.003s
I20260928 23:43:43.614167 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34431
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:43.727378 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:43.727669 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:43.727715 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:43.731891 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:43.732012 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:43.732079 22855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:43:43.737767 22855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:43.738828 22855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:43.739908 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:43.750754 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:43.750748 22861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:43.750862 22863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:43.751134 22855 server_base.cc:1034] running on GCE node
I20260928 23:43:43.751590 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:43.752301 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:43.753494 22855 hybrid_clock.cc:648] HybridClock initialized: now 1790639023753449 us; error 63 us; skew 500 ppm
Sep 28 23:43:43 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639023, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:43.756834 22855 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:43:43.757931 22855 webserver.cc:460] Webserver started at http://127.21.112.67:42693/ using document root <none> and password file <none>
I20260928 23:43:43.758530 22855 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:43.758594 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:43.758781 22855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:43.760951 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "5ff829d43e314983a4ebc5f347d4fabd"
format_stamp: "Formatted at 2026-09-28 23:43:43 on dist-test-slave-c3s8"
server_key: "48a5350593a48db502cf63878eeed655"
server_key_iv: "34cc795716fc65c1f1113deb45e39384"
server_key_version: "encryptionkey@0"
I20260928 23:43:43.761471 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "5ff829d43e314983a4ebc5f347d4fabd"
format_stamp: "Formatted at 2026-09-28 23:43:43 on dist-test-slave-c3s8"
server_key: "48a5350593a48db502cf63878eeed655"
server_key_iv: "34cc795716fc65c1f1113deb45e39384"
server_key_version: "encryptionkey@0"
I20260928 23:43:43.766074 22855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:43:43.768774 22870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:43.770229 22855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:43.770370 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "5ff829d43e314983a4ebc5f347d4fabd"
format_stamp: "Formatted at 2026-09-28 23:43:43 on dist-test-slave-c3s8"
server_key: "48a5350593a48db502cf63878eeed655"
server_key_iv: "34cc795716fc65c1f1113deb45e39384"
server_key_version: "encryptionkey@0"
I20260928 23:43:43.770495 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:43.804641 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:43.808166 22855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:43.808382 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:43.809005 22855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:43.809955 22855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:43.810004 22855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:43.810068 22855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:43.810103 22855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:43.819010 22855 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:40903
I20260928 23:43:43.819031 22983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:40903 every 8 connection(s)
I20260928 23:43:43.820008 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:43:43.821004 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 22855
I20260928 23:43:43.821130 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:43:43.821416 21953 external_mini_cluster.cc:1428] Setting key 628f1f2fb98ea79f28e549ada4c4fc7f
Sep 28 23:43:43 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639023, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:43.835206 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.822198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:50685 (local address 127.21.112.126:34431)
0928 23:43:43.822470 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.822474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.823323 (+ 849us) server_negotiation.cc:407] Connection header received
0928 23:43:43.824771 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.824774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.824822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.824907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:43.826257 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.826826 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.828077 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.828310 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.828396 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.831204 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:43.831207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:43.831210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:43.831244 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:43.832797 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.833345 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.833347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.833348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.833427 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.833875 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.833878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.833879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.834067 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:43.834209 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.834865 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.835002 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260928 23:43:43.836192 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.822562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34431 (local address 127.21.112.67:50685)
0928 23:43:43.823156 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.823216 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.824553 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.825174 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.825192 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.825593 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:43.826113 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.826126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.826953 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.826955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.827949 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.827959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.828452 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.828454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.828603 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.829262 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:43.829279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:43.831030 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:43.832958 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:43.832963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:43.832976 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:43.833251 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:43.833515 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:43.833517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:43.833518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:43.833743 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:43.834201 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:43.834207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:43.834549 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.834754 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.835124 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":164,"threads_started":1}
I20260928 23:43:43.837311 22984 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34431
I20260928 23:43:43.837529 22984 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:43.838035 22984 heartbeater.cc:511] Master 127.21.112.126:34431 requested a full tablet report, sending...
I20260928 23:43:43.839017 22523 ts_manager.cc:194] Registered new tserver with Master: 5ff829d43e314983a4ebc5f347d4fabd (127.21.112.67:40903)
I20260928 23:43:43.839572 22523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:50685
I20260928 23:43:43.847909 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:43:43.859812 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.851806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.21.112.126:34431)
0928 23:43:43.852011 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.852016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.852098 (+ 82us) server_negotiation.cc:407] Connection header received
0928 23:43:43.852223 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.852225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.852271 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.852342 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:43.853252 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.853829 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.854644 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.854897 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.854977 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.856627 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:43.856630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:43.856633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:43.856660 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:43.858087 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.858548 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.858550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.858552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.858630 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.858879 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.858881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.858882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.859111 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:43.859221 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.859471 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.859583 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 23:43:43.865573 22523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:43:43.868685 22523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:43.884276 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.879108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.21.112.66:36071)
0928 23:43:43.879992 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.879998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.880017 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:43:43.880078 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.880081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.880231 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.880381 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:43.880604 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.881612 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.882887 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.883545 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.883672 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.883833 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.884009 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.884122 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":131,"threads_started":1}
I20260928 23:43:43.884310 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.878948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:36071 (local address 127.0.0.1:38034)
0928 23:43:43.879518 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.879533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.879619 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.880357 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.880360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.880378 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:43.880490 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.880499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.881750 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.881756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.882741 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.882751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.883672 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.883675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.883798 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.883835 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.883951 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.884086 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":27,"spinlock_wait_cycles":2816,"thread_start_us":91,"threads_started":1}
I20260928 23:43:43.884773 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.878187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.21.112.67:40903)
0928 23:43:43.878613 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.878624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.878650 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:43:43.878971 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.878978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.879198 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.879361 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:43.879946 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.881080 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.882843 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.883599 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.883707 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.883840 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.884529 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.884628 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":203,"threads_started":1}
I20260928 23:43:43.884824 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.879268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40359 (local address 127.0.0.1:44716)
0928 23:43:43.879652 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.879666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.879764 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.880525 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.880531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.880552 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:43.880652 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.880657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.881935 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.881939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.883059 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.883068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.884318 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.884322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.884510 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.884530 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.884620 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.884694 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":84,"threads_started":1}
I20260928 23:43:43.884311 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.878079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:40903 (local address 127.0.0.1:55930)
0928 23:43:43.878523 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.878587 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.878732 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.879572 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.879580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.879604 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:43.879748 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.879762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.881230 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.881234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.882707 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.882718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.883913 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.883917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.884002 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.884021 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.884085 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.884134 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260928 23:43:43.885187 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.879659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.21.112.65:40359)
0928 23:43:43.879964 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.879971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.879996 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:43:43.880066 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.880071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.880266 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.880422 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:43.880769 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.881799 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.883183 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.884023 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.884124 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.884251 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.884665 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.884758 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
I20260928 23:43:43.888017 22648 tablet_service.cc:1467] Processing CreateTablet for tablet ea2176e6cb6b434db8514bd3c7d18602 (DEFAULT_TABLE table=test-table [id=67ff98f3f5ba482e94809e0ef6d7e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:43.888042 22918 tablet_service.cc:1467] Processing CreateTablet for tablet ea2176e6cb6b434db8514bd3c7d18602 (DEFAULT_TABLE table=test-table [id=67ff98f3f5ba482e94809e0ef6d7e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:43.888042 22783 tablet_service.cc:1467] Processing CreateTablet for tablet ea2176e6cb6b434db8514bd3c7d18602 (DEFAULT_TABLE table=test-table [id=67ff98f3f5ba482e94809e0ef6d7e6dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:43.889137 22918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2176e6cb6b434db8514bd3c7d18602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:43.889130 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2176e6cb6b434db8514bd3c7d18602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:43.889139 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2176e6cb6b434db8514bd3c7d18602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:43.895998 23003 tablet_bootstrap.cc:492] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62: Bootstrap starting.
I20260928 23:43:43.896724 23004 tablet_bootstrap.cc:492] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd: Bootstrap starting.
I20260928 23:43:43.899314 23004 tablet_bootstrap.cc:654] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:43.899652 23003 tablet_bootstrap.cc:654] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:43.900694 23004 log.cc:824] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:43.900705 23003 log.cc:824] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:43.901501 23005 tablet_bootstrap.cc:492] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: Bootstrap starting.
I20260928 23:43:43.903010 23003 tablet_bootstrap.cc:492] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62: No bootstrap required, opened a new log
I20260928 23:43:43.903038 23004 tablet_bootstrap.cc:492] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd: No bootstrap required, opened a new log
I20260928 23:43:43.903230 23003 ts_tablet_manager.cc:1397] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:43:43.903230 23004 ts_tablet_manager.cc:1397] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:43:43.903944 23005 tablet_bootstrap.cc:654] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:43.904855 23005 log.cc:824] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:43.907157 23005 tablet_bootstrap.cc:492] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: No bootstrap required, opened a new log
I20260928 23:43:43.907383 23005 ts_tablet_manager.cc:1397] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:43:43.907155 23003 raft_consensus.cc:348] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.907480 23003 raft_consensus.cc:374] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:43.907554 23003 raft_consensus.cc:729] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a75f37089ed44e4ba1d41ea38354f62, State: Initialized, Role: FOLLOWER
I20260928 23:43:43.907537 23004 raft_consensus.cc:348] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.907902 23004 raft_consensus.cc:374] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:43.907970 23004 raft_consensus.cc:729] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff829d43e314983a4ebc5f347d4fabd, State: Initialized, Role: FOLLOWER
I20260928 23:43:43.908075 23003 consensus_queue.cc:260] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.908443 23004 consensus_queue.cc:260] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.909168 23003 ts_tablet_manager.cc:1428] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:43:43.909603 23004 ts_tablet_manager.cc:1428] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 23:43:43.909977 22984 heartbeater.cc:503] Master 127.21.112.126:34431 was elected leader, sending a full tablet report...
I20260928 23:43:43.911924 23005 raft_consensus.cc:348] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.912304 23005 raft_consensus.cc:374] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:43.912410 23005 raft_consensus.cc:729] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36925091951c4f639df1218bc52f102d, State: Initialized, Role: FOLLOWER
I20260928 23:43:43.912900 23005 consensus_queue.cc:260] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.913667 22849 heartbeater.cc:503] Master 127.21.112.126:34431 was elected leader, sending a full tablet report...
I20260928 23:43:43.914072 23005 ts_tablet_manager.cc:1428] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:43:43.956396 23009 raft_consensus.cc:482] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:43.956636 23009 raft_consensus.cc:504] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.957963 23009 leader_election.cc:288] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff829d43e314983a4ebc5f347d4fabd (127.21.112.67:40903), 36925091951c4f639df1218bc52f102d (127.21.112.66:36071)
I20260928 23:43:43.963006 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.958939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:56665 (local address 127.21.112.66:36071)
0928 23:43:43.959094 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.959099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.959142 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:43:43.959280 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.959282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.959323 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.959390 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:43.959822 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.960605 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.961957 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.962422 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.962552 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.962607 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.962790 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.962870 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:43:43.963025 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.958716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:36071 (local address 127.21.112.65:56665)
0928 23:43:43.959072 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.959087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.959172 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.959479 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.959484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.959510 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:43.959683 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.959691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.960731 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.960736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.961832 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.961842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.962552 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.962555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.962651 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.962675 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.962771 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.962846 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260928 23:43:43.963064 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.958342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:55345 (local address 127.21.112.67:40903)
0928 23:43:43.958513 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.958518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.958709 (+ 191us) server_negotiation.cc:407] Connection header received
0928 23:43:43.958910 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.958914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.958992 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.959084 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:43.959655 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.960491 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.961943 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.962483 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.962612 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.962680 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.962846 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.962915 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:43:43.963071 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.958261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:40903 (local address 127.21.112.65:55345)
0928 23:43:43.958608 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:43.958621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:43.958769 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:43.959114 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:43.959120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:43.959144 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:43.959437 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.959456 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.960618 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.960628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.961818 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:43.961829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.962601 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.962604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:43.962712 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.962742 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:43:43.962832 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:43:43.962904 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":44,"thread_start_us":142,"threads_started":1}
W20260928 23:43:43.963860 22850 tablet.cc:2369] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:43:43.963860 22938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2176e6cb6b434db8514bd3c7d18602" candidate_uuid: "5a75f37089ed44e4ba1d41ea38354f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff829d43e314983a4ebc5f347d4fabd" is_pre_election: true
I20260928 23:43:43.964030 22802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2176e6cb6b434db8514bd3c7d18602" candidate_uuid: "5a75f37089ed44e4ba1d41ea38354f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36925091951c4f639df1218bc52f102d" is_pre_election: true
I20260928 23:43:43.964203 22938 raft_consensus.cc:2454] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a75f37089ed44e4ba1d41ea38354f62 in term 0.
I20260928 23:43:43.964282 22802 raft_consensus.cc:2454] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a75f37089ed44e4ba1d41ea38354f62 in term 0.
I20260928 23:43:43.964816 22601 leader_election.cc:302] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36925091951c4f639df1218bc52f102d, 5a75f37089ed44e4ba1d41ea38354f62; no voters:
I20260928 23:43:43.965150 23009 raft_consensus.cc:2781] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:43.965231 23009 raft_consensus.cc:482] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:43.965262 23009 raft_consensus.cc:3037] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:43.966415 23009 raft_consensus.cc:504] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.966782 23009 leader_election.cc:288] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff829d43e314983a4ebc5f347d4fabd (127.21.112.67:40903), 36925091951c4f639df1218bc52f102d (127.21.112.66:36071)
I20260928 23:43:43.967195 22938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2176e6cb6b434db8514bd3c7d18602" candidate_uuid: "5a75f37089ed44e4ba1d41ea38354f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff829d43e314983a4ebc5f347d4fabd"
I20260928 23:43:43.967276 22802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2176e6cb6b434db8514bd3c7d18602" candidate_uuid: "5a75f37089ed44e4ba1d41ea38354f62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36925091951c4f639df1218bc52f102d"
I20260928 23:43:43.967329 22938 raft_consensus.cc:3037] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:43.967391 22802 raft_consensus.cc:3037] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:43.968618 22938 raft_consensus.cc:2454] T ea2176e6cb6b434db8514bd3c7d18602 P 5ff829d43e314983a4ebc5f347d4fabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a75f37089ed44e4ba1d41ea38354f62 in term 1.
I20260928 23:43:43.968618 22802 raft_consensus.cc:2454] T ea2176e6cb6b434db8514bd3c7d18602 P 36925091951c4f639df1218bc52f102d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a75f37089ed44e4ba1d41ea38354f62 in term 1.
I20260928 23:43:43.968951 22601 leader_election.cc:302] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a75f37089ed44e4ba1d41ea38354f62, 5ff829d43e314983a4ebc5f347d4fabd; no voters:
I20260928 23:43:43.969154 23009 raft_consensus.cc:2781] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:43.969412 23009 raft_consensus.cc:686] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [term 1 LEADER]: Becoming Leader. State: Replica: 5a75f37089ed44e4ba1d41ea38354f62, State: Running, Role: LEADER
I20260928 23:43:43.969830 23009 consensus_queue.cc:237] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } }
I20260928 23:43:43.973366 22523 catalog_manager.cc:5495] T ea2176e6cb6b434db8514bd3c7d18602 P 5a75f37089ed44e4ba1d41ea38354f62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a75f37089ed44e4ba1d41ea38354f62 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "5a75f37089ed44e4ba1d41ea38354f62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff829d43e314983a4ebc5f347d4fabd" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36925091951c4f639df1218bc52f102d" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 36071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a75f37089ed44e4ba1d41ea38354f62" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40359 } health_report { overall_health: HEALTHY } } }
Sep 28 23:43:43 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:43:43.991572 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:43.982339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.21.112.65:40359)
0928 23:43:43.982494 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:43.982498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:43.982584 (+ 86us) server_negotiation.cc:407] Connection header received
0928 23:43:43.982681 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:43.982683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:43.982724 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:43.982795 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:43.983513 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.984029 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.984956 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:43.985174 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:43.985250 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:43.987999 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:43.988012 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:43.988027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:43.988050 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:43.989750 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.990207 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.990214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.990215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.990295 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:43.990604 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:43.990608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:43.990610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:43.990864 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:43.991085 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:43.991244 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:43:43.991380 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 23:43:43 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:43:44 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:43:44.021607 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:44.010708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.21.112.65:40359)
0928 23:43:44.010926 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:44.010932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:44.011091 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:43:44.011265 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:44.011268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:44.011341 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:44.011446 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:44.012425 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:44.013392 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:44.014776 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:44.015124 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:44.015249 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:44.018165 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:44.018169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:44.018171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:44.018202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:44.019784 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:44.020194 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:44.020197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:44.020198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:44.020289 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:44.020567 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:44.020570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:44.020571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:44.020753 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:44.020896 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:44.021130 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 23:43:44.021237 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 23:43:44 dist-test-slave-c3s8 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:43:44 dist-test-slave-c3s8 krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:43:44.049244 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:44.039839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.21.112.65:40359)
0928 23:43:44.040022 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:44.040027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:44.040172 (+ 145us) server_negotiation.cc:407] Connection header received
0928 23:43:44.040369 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:44.040372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:44.040417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:44.040508 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:44.041227 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:44.041800 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:44.042676 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:44.042907 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:44.043023 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:44.045906 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:44.045909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:44.045911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:44.045958 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:44.047340 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:44.047863 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:44.047867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:44.047868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:44.047994 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:44.048310 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:44.048313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:44.048315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:44.048506 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:44.048656 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:44.048936 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 23:43:44.049063 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 23:43:44.050768 22628 tablet_service.cc:2908] Rejecting scan request for tablet ea2176e6cb6b434db8514bd3c7d18602: Uninitialized: clean time has not yet been initialized
I20260928 23:43:44.053074 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22585
I20260928 23:43:44.059419 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22720
I20260928 23:43:44.064925 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22855
I20260928 23:43:44.070789 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 22491
2026-09-28T23:43:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6770 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:44 dist-test-slave-c3s8 krb5kdc[23028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:44 dist-test-slave-c3s8 krb5kdc[23028](info): set up 2 sockets
Sep 28 23:43:44 dist-test-slave-c3s8 krb5kdc[23028](info): commencing operation
krb5kdc: starting...
W20260928 23:43:45.117848 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:43:45.252712 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:40649
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:45.365262 23041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:45.365628 23041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:45.365751 23041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:45.369830 23041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:45.369932 23041 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:43:45.369966 23041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:45.370011 23041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:45.370048 23041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:45.370085 23041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:45.375546 23041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:40649
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:45.376715 23041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:45.377805 23041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:45.386528 23049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:45.386536 23047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:45.386601 23046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:45.386942 23041 server_base.cc:1034] running on GCE node
I20260928 23:43:45.387547 23041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:45.388613 23041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:45.389804 23041 hybrid_clock.cc:648] HybridClock initialized: now 1790639025389790 us; error 26 us; skew 500 ppm
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:45.392515 23041 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:45.393455 23041 webserver.cc:460] Webserver started at http://127.21.112.126:34295/ using document root <none> and password file <none>
I20260928 23:43:45.394037 23041 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:45.394099 23041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:45.394260 23041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:45.396234 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "9e9e0d8c92644348b184f229dc04295f"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "783ac75a6189a681c81590511049274d"
server_key_iv: "93ba0761743feae775e9f2528bbdfcab"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.396751 23041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "9e9e0d8c92644348b184f229dc04295f"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "783ac75a6189a681c81590511049274d"
server_key_iv: "93ba0761743feae775e9f2528bbdfcab"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.400467 23041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 23:43:45.402868 23056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:45.404170 23041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:43:45.404305 23041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "9e9e0d8c92644348b184f229dc04295f"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "783ac75a6189a681c81590511049274d"
server_key_iv: "93ba0761743feae775e9f2528bbdfcab"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.404398 23041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:45.419855 23041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:45.422932 23041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:45.423091 23041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:45.431674 23041 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:40649
I20260928 23:43:45.431701 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:40649 every 8 connection(s)
I20260928 23:43:45.432673 23041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:45.435724 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:45.439586 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23041
I20260928 23:43:45.439709 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:45.439973 21953 external_mini_cluster.cc:1428] Setting key 5210ed704ba38cabe23fba7b3a630d67
I20260928 23:43:45.442015 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: Bootstrap starting.
I20260928 23:43:45.444403 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:45.445390 23119 log.cc:824] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:45.447651 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: No bootstrap required, opened a new log
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:45.452116 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9e0d8c92644348b184f229dc04295f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40649 } }
I20260928 23:43:45.452454 23119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:45.452531 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9e0d8c92644348b184f229dc04295f, State: Initialized, Role: FOLLOWER
I20260928 23:43:45.453120 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9e0d8c92644348b184f229dc04295f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40649 } }
I20260928 23:43:45.453306 23119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:45.453387 23119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:45.453498 23119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:45.454864 23119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9e0d8c92644348b184f229dc04295f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40649 } }
I20260928 23:43:45.455276 23119 leader_election.cc:302] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e9e0d8c92644348b184f229dc04295f; no voters:
I20260928 23:43:45.455703 23119 leader_election.cc:288] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:45.455931 23124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:45.456172 23124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [term 1 LEADER]: Becoming Leader. State: Replica: 9e9e0d8c92644348b184f229dc04295f, State: Running, Role: LEADER
I20260928 23:43:45.456578 23124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9e0d8c92644348b184f229dc04295f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40649 } }
I20260928 23:43:45.456856 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.441562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.21.112.126:40649)
0928 23:43:45.442060 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:45.442111 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:45.442164 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:43:45.442923 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:45.442955 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:45.443335 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:45.443724 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:45.444658 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.446132 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.447101 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.447794 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.447968 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.450912 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:45.450920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:45.450939 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:45.450974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:45.453158 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.453861 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.453867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.453873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.454024 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.454335 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.454338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.454340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.454944 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:45.455141 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:45.455476 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 23:43:45.455810 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260928 23:43:45.457127 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:43:45.458467 23126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9e0d8c92644348b184f229dc04295f. Latest consensus state: current_term: 1 leader_uuid: "9e9e0d8c92644348b184f229dc04295f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9e0d8c92644348b184f229dc04295f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40649 } } }
I20260928 23:43:45.458590 23126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:45.458931 23125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e9e0d8c92644348b184f229dc04295f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9e0d8c92644348b184f229dc04295f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40649 } } }
I20260928 23:43:45.459015 23125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f [sys.catalog]: This master's current role is: LEADER
W20260928 23:43:45.462570 23139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:43:45.462652 23139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:43:45.462829 23133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:45.463621 23133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:45.470376 23133 catalog_manager.cc:1347] Generated new cluster ID: dae0aa96603f4a1cafbb6fda2e35b383
I20260928 23:43:45.470476 23133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:45.490327 23133 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:45.491668 23133 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:45.498528 23133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: Generated new TSK 0
I20260928 23:43:45.499379 23133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:43:45.529160 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:40649
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:45.645896 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:45.646199 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:45.646247 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:45.650573 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:45.650653 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:45.650694 23145 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:43:45.650753 23145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:43:45.656417 23145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:45.657788 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:45.659018 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:45.669185 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:45.669185 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:45.669283 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:45.669945 23145 server_base.cc:1034] running on GCE node
I20260928 23:43:45.670444 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:45.671264 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:45.672479 23145 hybrid_clock.cc:648] HybridClock initialized: now 1790639025672403 us; error 90 us; skew 500 ppm
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:45.675786 23145 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:43:45.676862 23145 webserver.cc:460] Webserver started at http://127.21.112.65:34041/ using document root <none> and password file <none>
I20260928 23:43:45.677449 23145 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:45.677515 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:45.677788 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:45.680065 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "1253aad682fc49b9b3475270df2448dc"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "93485916599f9afc7597035e01ff6415"
server_key_iv: "fb6f8743a4c3139b07eaa5b7129f8d1e"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.680639 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "1253aad682fc49b9b3475270df2448dc"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "93485916599f9afc7597035e01ff6415"
server_key_iv: "fb6f8743a4c3139b07eaa5b7129f8d1e"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.684944 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:43:45.688128 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:45.689683 23145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:45.689821 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "1253aad682fc49b9b3475270df2448dc"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "93485916599f9afc7597035e01ff6415"
server_key_iv: "fb6f8743a4c3139b07eaa5b7129f8d1e"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.689941 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:45.737425 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:45.741365 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:45.741634 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:45.743257 23145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:45.743310 23145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:45.743355 23145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:45.743418 23145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:45.756130 23145 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:40273
I20260928 23:43:45.756933 23275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:40273 every 8 connection(s)
I20260928 23:43:45.757299 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:43:45.759245 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.744294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:56435 (local address 127.21.112.126:40649)
0928 23:43:45.744464 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:45.744470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:45.745370 (+ 900us) server_negotiation.cc:407] Connection header received
0928 23:43:45.746474 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:45.746478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:45.746536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:45.746599 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:45.748385 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.749126 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.750718 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.751193 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.751363 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.754945 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:45.754949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:45.754952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:45.754987 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:45.756691 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.757356 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.757359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.757361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.757463 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.757880 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.757883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.757885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.758195 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:45.758316 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:45.758925 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 23:43:45.759060 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:43:45.760061 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.744706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.21.112.65:56435)
0928 23:43:45.745186 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:45.745239 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:45.746197 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:45.747090 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:45.747106 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:45.747537 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:45.748198 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:45.748223 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.749248 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.749252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:45.750555 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:45.750567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.751432 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.751437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:45.751684 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.752425 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:45.752443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:45.754731 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:45.756830 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:45.756839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:45.756857 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:45.757235 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:45.757601 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:45.757606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:45.757608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:45.757767 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:45.758313 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:45.758320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:45.758642 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 23:43:45.758866 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 23:43:45.759209 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":142,"threads_started":1}
I20260928 23:43:45.761417 23276 heartbeater.cc:347] Connected to a master server at 127.21.112.126:40649
I20260928 23:43:45.761612 23276 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:45.762140 23276 heartbeater.cc:511] Master 127.21.112.126:40649 requested a full tablet report, sending...
I20260928 23:43:45.763724 23072 ts_manager.cc:194] Registered new tserver with Master: 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273)
I20260928 23:43:45.765149 23072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:56435
I20260928 23:43:45.766845 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23145
I20260928 23:43:45.766966 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:43:45.767241 21953 external_mini_cluster.cc:1428] Setting key b962733c73b5b0d65fbd29742bd54e3f
I20260928 23:43:45.771585 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.763731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.21.112.65:38075)
0928 23:43:45.764144 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:45.764156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:45.764247 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:45.764469 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:45.764471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:45.764640 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:45.764740 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:45.764744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.765675 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.765679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:45.766330 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:45.766338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.766817 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.766821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:45.766984 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.767611 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:45.767626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:45.768038 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:45.769893 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:45.769899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:45.769904 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:45.770175 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:45.770517 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:45.770525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:45.770526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:45.770605 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:45.771146 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:45.771148 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:45.771234 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 23:43:45.771320 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:43:45.771436 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":116,"threads_started":1}
I20260928 23:43:45.771795 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.763861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:38075 (local address 127.21.112.126:40649)
0928 23:43:45.764011 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:45.764014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:45.764241 (+ 227us) server_negotiation.cc:407] Connection header received
0928 23:43:45.764325 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:45.764328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:45.764371 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:45.764465 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:45.764847 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.765559 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.766452 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.766698 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.766812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.768171 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:45.768175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:45.768178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:45.768202 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:45.769782 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.770290 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.770298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.770304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.770408 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.770696 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.770701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.770704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.771040 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:45.771151 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:45.771419 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 23:43:45.771554 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:43:45.796502 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:40649
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:45.910969 23285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:45.911242 23285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:45.911288 23285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:45.915508 23285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:45.915575 23285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:45.915616 23285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:43:45.915696 23285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:43:45.921103 23285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:45.922220 23285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:45.923278 23285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:45.933349 23291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:45.933269 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:45.933413 23285 server_base.cc:1034] running on GCE node
W20260928 23:43:45.933626 23293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:45.934108 23285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:45.934711 23285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:45.935920 23285 hybrid_clock.cc:648] HybridClock initialized: now 1790639025935901 us; error 40 us; skew 500 ppm
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:45.939178 23285 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:43:45.940219 23285 webserver.cc:460] Webserver started at http://127.21.112.66:42055/ using document root <none> and password file <none>
I20260928 23:43:45.940857 23285 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:45.940955 23285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:45.941196 23285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:45.943670 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "87045fb026e94528a3f48ff81583ee00"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "dc715c5e369725638f5edbb4d1f3aa76"
server_key_iv: "9a99d85f4a094477fae0cc22f857053c"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.944437 23285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "87045fb026e94528a3f48ff81583ee00"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "dc715c5e369725638f5edbb4d1f3aa76"
server_key_iv: "9a99d85f4a094477fae0cc22f857053c"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.948431 23285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 23:43:45.951159 23300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:45.952517 23285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:43:45.952688 23285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "87045fb026e94528a3f48ff81583ee00"
format_stamp: "Formatted at 2026-09-28 23:43:45 on dist-test-slave-c3s8"
server_key: "dc715c5e369725638f5edbb4d1f3aa76"
server_key_iv: "9a99d85f4a094477fae0cc22f857053c"
server_key_version: "encryptionkey@0"
I20260928 23:43:45.952824 23285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:45.966730 23285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:45.970388 23285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:45.970613 23285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:45.972681 23285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:45.972977 23285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:45.973052 23285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:45.973075 23285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:45.986094 23285 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:40749
I20260928 23:43:45.987310 23415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:40749 every 8 connection(s)
I20260928 23:43:45.987588 23285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:43:45 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:45.993561 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23285
I20260928 23:43:45.993683 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:43:45.994040 21953 external_mini_cluster.cc:1428] Setting key f65b76741cbd0f49a574f19efbd9805c
I20260928 23:43:45.996035 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.973694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:40577 (local address 127.21.112.126:40649)
0928 23:43:45.973862 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:45.973867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:45.978744 (+ 4877us) server_negotiation.cc:407] Connection header received
0928 23:43:45.980706 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:45.980713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:45.980800 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:45.980874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:45.984193 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.985095 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.987143 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.987503 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.987619 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.991105 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:45.991109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:45.991112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:45.991142 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:45.992880 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.993603 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.993608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.993611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.993737 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:45.994655 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:45.994659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:45.994660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:45.994916 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:45.995020 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:45.995725 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 23:43:45.995861 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:43:45.997298 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:45.977583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.21.112.66:40577)
0928 23:43:45.978243 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:45.978311 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:45.980374 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:45.981621 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:45.981634 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:45.982202 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:45.983980 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:45.984004 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.985298 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.985301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:45.986994 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:45.987026 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:45.987675 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:45.987679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:45.987907 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:45.988718 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:45.988737 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:45.990885 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:45.993022 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:45.993027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:45.993040 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:45.993425 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:45.994156 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:45.994160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:45.994162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:45.994362 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:45.995024 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:45.995031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:45.995409 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 23:43:45.995641 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 23:43:45.996058 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":157,"threads_started":1}
I20260928 23:43:46.001338 23416 heartbeater.cc:347] Connected to a master server at 127.21.112.126:40649
I20260928 23:43:46.001616 23416 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:46.002213 23416 heartbeater.cc:511] Master 127.21.112.126:40649 requested a full tablet report, sending...
I20260928 23:43:46.003906 23072 ts_manager.cc:194] Registered new tserver with Master: 87045fb026e94528a3f48ff81583ee00 (127.21.112.66:40749)
I20260928 23:43:46.004776 23072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:40577
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:43:46.019868 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.005521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:44035 (local address 127.21.112.126:40649)
0928 23:43:46.005689 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.005694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.009963 (+ 4269us) server_negotiation.cc:407] Connection header received
0928 23:43:46.010113 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.010119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.010187 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.010299 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.010937 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.011854 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.013235 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.013704 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.013852 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.015489 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.015497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.015503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.015537 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.017508 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.018133 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.018139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.018143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.018246 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.018637 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.018642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.018647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.019019 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.019104 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.019535 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.019678 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 23:43:46.020898 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.005552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.21.112.66:44035)
0928 23:43:46.009841 (+ 4289us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.009861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.010002 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.010359 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.010363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.010628 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:46.010808 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.010819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.012001 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.012005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.013083 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.013096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.013871 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.013878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.014011 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.014696 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:46.014718 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:46.015262 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:46.017670 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.017677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.017680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.018003 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.018391 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.018397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.018401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.018510 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.019184 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:46.019190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:46.019318 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.020418 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.020651 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4150,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:43:46.032186 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:40649
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:46.152990 23425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:46.153399 23425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:46.153492 23425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:46.158740 23425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:46.158818 23425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:46.158895 23425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:43:46.158967 23425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:43:46.165761 23425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:46.167081 23425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:46.168377 23425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:46.179819 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:46.179962 23425 server_base.cc:1034] running on GCE node
W20260928 23:43:46.179824 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:46.179824 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:46.180685 23425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:46.181449 23425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:46.182662 23425 hybrid_clock.cc:648] HybridClock initialized: now 1790639026182628 us; error 54 us; skew 500 ppm
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639026, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:46.192548 23425 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:43:46.193975 23425 webserver.cc:460] Webserver started at http://127.21.112.67:45209/ using document root <none> and password file <none>
I20260928 23:43:46.194690 23425 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:46.194756 23425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:46.194995 23425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:46.197094 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "cb5ebf18bac6425bb22c06567f6e543c"
format_stamp: "Formatted at 2026-09-28 23:43:46 on dist-test-slave-c3s8"
server_key: "622f47e27fef7e1709f0c453f1fa5e4e"
server_key_iv: "fa462f5ec50526ab3058888abfff0e01"
server_key_version: "encryptionkey@0"
I20260928 23:43:46.197695 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "cb5ebf18bac6425bb22c06567f6e543c"
format_stamp: "Formatted at 2026-09-28 23:43:46 on dist-test-slave-c3s8"
server_key: "622f47e27fef7e1709f0c453f1fa5e4e"
server_key_iv: "fa462f5ec50526ab3058888abfff0e01"
server_key_version: "encryptionkey@0"
I20260928 23:43:46.201776 23425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:43:46.204339 23440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:46.205969 23425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:46.206106 23425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "cb5ebf18bac6425bb22c06567f6e543c"
format_stamp: "Formatted at 2026-09-28 23:43:46 on dist-test-slave-c3s8"
server_key: "622f47e27fef7e1709f0c453f1fa5e4e"
server_key_iv: "fa462f5ec50526ab3058888abfff0e01"
server_key_version: "encryptionkey@0"
I20260928 23:43:46.206228 23425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:46.220889 23425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:46.224267 23425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:46.224480 23425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:46.226168 23425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:46.226218 23425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:46.226260 23425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:46.226296 23425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639026, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:46.237581 23425 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:34067
I20260928 23:43:46.237937 23555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:34067 every 8 connection(s)
I20260928 23:43:46.238700 23425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:43:46.240831 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.226978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:47741 (local address 127.21.112.126:40649)
0928 23:43:46.227132 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.227137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.228156 (+ 1019us) server_negotiation.cc:407] Connection header received
0928 23:43:46.229589 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.229594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.229663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.229799 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.231180 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.231837 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.233022 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.233254 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.233384 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.236647 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.236651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.236653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.236700 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.238420 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.239012 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.239015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.239017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.239107 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.239688 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.239690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.239691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.239885 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.239997 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.240495 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.240630 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:43:46.241627 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23425
I20260928 23:43:46.241755 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:43:46.241813 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.227437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.21.112.67:47741)
0928 23:43:46.227955 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.228020 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.229223 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.229984 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.229993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.230358 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:46.231027 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.231045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.231974 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.231977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.232892 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.232901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.233437 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.233445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.233731 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.234499 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:46.234518 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:46.236440 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:46.238550 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.238556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.238570 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.238904 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.239245 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.239248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.239250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.239427 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.240005 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:46.240013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:46.240256 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.240450 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.240807 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260928 23:43:46.242036 21953 external_mini_cluster.cc:1428] Setting key 48056dc855c5543d23daee79dbd07464
I20260928 23:43:46.243523 23556 heartbeater.cc:347] Connected to a master server at 127.21.112.126:40649
I20260928 23:43:46.243678 23556 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:46.244299 23556 heartbeater.cc:511] Master 127.21.112.126:40649 requested a full tablet report, sending...
I20260928 23:43:46.245473 23071 ts_manager.cc:194] Registered new tserver with Master: cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.246258 23071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:47741
I20260928 23:43:46.253706 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.246579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:60965 (local address 127.21.112.126:40649)
0928 23:43:46.246694 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.246697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.246708 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:43:46.246738 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.246740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.246772 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.246869 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.247299 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.248114 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.248971 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.249184 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.249267 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.250411 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.250415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.250417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.250440 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.251954 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.252451 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.252454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.252455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.252536 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.252842 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.252846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.252847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.253058 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.253149 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.253375 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.253486 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:43:46.253710 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.245693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.21.112.67:60965)
0928 23:43:46.246027 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.246042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.246123 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.246895 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.246897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.247080 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:46.247186 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.247191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.248248 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.248251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.248857 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.248866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.249287 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.249289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.249376 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.249894 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:46.249906 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:46.250267 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:46.252106 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.252109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.252110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.252344 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.252625 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.252628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.252629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.252726 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.253151 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:46.253153 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:46.253239 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.253337 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.253473 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
I20260928 23:43:46.256158 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:46.284042 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.275411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.21.112.126:40649)
0928 23:43:46.275567 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.275571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.275583 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:43:46.275629 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.275632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.275671 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.275767 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.276546 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.277083 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.278002 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.278231 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.278354 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.280874 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.280877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.280879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.280914 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.282311 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.282776 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.282781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.282783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.282868 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.283143 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.283147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.283149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.283349 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.283491 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.283743 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.283865 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:43:46.287603 23071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:43:46.290975 23071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:46.306844 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.300238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40273 (local address 127.0.0.1:51600)
0928 23:43:46.301483 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.301497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.301615 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.302498 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.302501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.302518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.302606 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.302612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.303634 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.303638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.305166 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.305176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.306273 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.306277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.306394 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.306435 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.306546 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.306661 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":59,"thread_start_us":126,"threads_started":1}
I20260928 23:43:46.307011 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.300324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.21.112.66:40749)
0928 23:43:46.300691 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.300703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.301172 (+ 469us) server_negotiation.cc:407] Connection header received
0928 23:43:46.301257 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.301264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.301522 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.301702 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.302173 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.303397 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.305338 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.306289 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.306390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.306547 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.306762 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.306855 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260928 23:43:46.307034 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.300977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:34067 (local address 127.0.0.1:53250)
0928 23:43:46.301722 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.301735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.301809 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.302539 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.302543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.302562 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.302666 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.302672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.304414 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.304418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.305670 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.305680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.306674 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.306680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.306764 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.306782 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.306839 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.306908 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":49,"thread_start_us":77,"threads_started":1}
I20260928 23:43:46.307313 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.300193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:40749 (local address 127.0.0.1:44380)
0928 23:43:46.300836 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.300890 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.301024 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.301904 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.301911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.301933 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.302066 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.302073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.303545 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.303549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.305191 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.305202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.306489 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.306493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.306592 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.306632 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.306707 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.306777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":186,"thread_start_us":82,"threads_started":1}
I20260928 23:43:46.307852 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.301356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.21.112.67:34067)
0928 23:43:46.302002 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.302010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.302064 (+ 54us) server_negotiation.cc:407] Connection header received
0928 23:43:46.302140 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.302145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.302350 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.302807 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.302933 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.304256 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.305801 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.306543 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.307384 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.307558 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.307606 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.307714 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":89,"threads_started":1}
I20260928 23:43:46.308564 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.301014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.21.112.65:40273)
0928 23:43:46.302038 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.302046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.302068 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:43:46.302141 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.302148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.302347 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.302529 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.302714 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.303432 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.305312 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.306136 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.306240 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.306402 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.306935 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.308416 (+ 1481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":92,"threads_started":1}
I20260928 23:43:46.310541 23347 tablet_service.cc:1467] Processing CreateTablet for tablet 4801be79c3af4653833ea40d2a7fe9c0 (DEFAULT_TABLE table=test-table [id=2d6ddf01e16d421998144c6cb1b83dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:46.310539 23490 tablet_service.cc:1467] Processing CreateTablet for tablet 4801be79c3af4653833ea40d2a7fe9c0 (DEFAULT_TABLE table=test-table [id=2d6ddf01e16d421998144c6cb1b83dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:46.310904 23210 tablet_service.cc:1467] Processing CreateTablet for tablet 4801be79c3af4653833ea40d2a7fe9c0 (DEFAULT_TABLE table=test-table [id=2d6ddf01e16d421998144c6cb1b83dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:46.311676 23490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4801be79c3af4653833ea40d2a7fe9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.311959 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4801be79c3af4653833ea40d2a7fe9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.312032 23347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4801be79c3af4653833ea40d2a7fe9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.317986 23579 tablet_bootstrap.cc:492] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Bootstrap starting.
I20260928 23:43:46.319538 23581 tablet_bootstrap.cc:492] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Bootstrap starting.
I20260928 23:43:46.320286 23580 tablet_bootstrap.cc:492] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Bootstrap starting.
I20260928 23:43:46.320470 23579 tablet_bootstrap.cc:654] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.321360 23579 log.cc:824] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:46.322610 23580 tablet_bootstrap.cc:654] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.323319 23580 log.cc:824] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:46.323719 23581 tablet_bootstrap.cc:654] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.324024 23579 tablet_bootstrap.cc:492] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: No bootstrap required, opened a new log
I20260928 23:43:46.324224 23579 ts_tablet_manager.cc:1397] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:43:46.324525 23581 log.cc:824] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:46.325361 23580 tablet_bootstrap.cc:492] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: No bootstrap required, opened a new log
I20260928 23:43:46.325582 23580 ts_tablet_manager.cc:1397] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:43:46.326571 23581 tablet_bootstrap.cc:492] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: No bootstrap required, opened a new log
I20260928 23:43:46.326752 23581 ts_tablet_manager.cc:1397] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:43:46.328518 23580 raft_consensus.cc:348] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.328766 23580 raft_consensus.cc:374] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.328815 23580 raft_consensus.cc:729] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87045fb026e94528a3f48ff81583ee00, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.328863 23579 raft_consensus.cc:348] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.329118 23579 raft_consensus.cc:374] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.329175 23579 raft_consensus.cc:729] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1253aad682fc49b9b3475270df2448dc, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.329304 23580 consensus_queue.cc:260] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.329614 23579 consensus_queue.cc:260] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.330330 23276 heartbeater.cc:503] Master 127.21.112.126:40649 was elected leader, sending a full tablet report...
I20260928 23:43:46.330591 23579 ts_tablet_manager.cc:1428] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:43:46.330595 23416 heartbeater.cc:503] Master 127.21.112.126:40649 was elected leader, sending a full tablet report...
I20260928 23:43:46.331234 23580 ts_tablet_manager.cc:1428] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:43:46.331526 23581 raft_consensus.cc:348] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.331826 23581 raft_consensus.cc:374] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.331910 23581 raft_consensus.cc:729] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5ebf18bac6425bb22c06567f6e543c, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.332379 23581 consensus_queue.cc:260] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.333101 23556 heartbeater.cc:503] Master 127.21.112.126:40649 was elected leader, sending a full tablet report...
I20260928 23:43:46.333480 23581 ts_tablet_manager.cc:1428] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:43:46.345882 23586 raft_consensus.cc:482] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:46.346118 23586 raft_consensus.cc:504] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.347273 23586 leader_election.cc:288] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87045fb026e94528a3f48ff81583ee00 (127.21.112.66:40749), cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.351915 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.347762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:33833 (local address 127.21.112.66:40749)
0928 23:43:46.347922 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.347927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.348084 (+ 157us) server_negotiation.cc:407] Connection header received
0928 23:43:46.348692 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.348698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.348759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.348803 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.349274 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.349869 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.350962 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.351356 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.351486 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.351546 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.351714 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.351788 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:43:46.351943 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.347647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:40749 (local address 127.21.112.65:33833)
0928 23:43:46.347994 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.348010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.348170 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.348978 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.348983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.349005 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.349155 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.349162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.349981 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.349987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.350829 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.350839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.351477 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.351481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.351579 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.351601 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.351699 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.351772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":3,"thread_start_us":140,"threads_started":1}
I20260928 23:43:46.352260 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.348265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:60415 (local address 127.21.112.67:34067)
0928 23:43:46.348409 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.348415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.348642 (+ 227us) server_negotiation.cc:407] Connection header received
0928 23:43:46.348888 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.348891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.348944 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.349020 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.349333 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.350026 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.351208 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.351649 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.351797 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.351848 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.352044 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.352119 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:43:46.352274 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.348158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:34067 (local address 127.21.112.65:60415)
0928 23:43:46.348529 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.348542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.348681 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.349051 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.349053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.349073 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.349197 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.349201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.350156 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.350163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.351063 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.351073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.351790 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.351794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.351909 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.351930 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.352028 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.352100 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":27,"thread_start_us":114,"threads_started":1}
I20260928 23:43:46.352771 23365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4801be79c3af4653833ea40d2a7fe9c0" candidate_uuid: "1253aad682fc49b9b3475270df2448dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87045fb026e94528a3f48ff81583ee00" is_pre_election: true
I20260928 23:43:46.352977 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4801be79c3af4653833ea40d2a7fe9c0" candidate_uuid: "1253aad682fc49b9b3475270df2448dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5ebf18bac6425bb22c06567f6e543c" is_pre_election: true
I20260928 23:43:46.353092 23365 raft_consensus.cc:2454] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1253aad682fc49b9b3475270df2448dc in term 0.
I20260928 23:43:46.353297 23510 raft_consensus.cc:2454] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1253aad682fc49b9b3475270df2448dc in term 0.
I20260928 23:43:46.353682 23161 leader_election.cc:302] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1253aad682fc49b9b3475270df2448dc, 87045fb026e94528a3f48ff81583ee00; no voters:
I20260928 23:43:46.353957 23586 raft_consensus.cc:2781] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:46.354053 23586 raft_consensus.cc:482] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:46.354099 23586 raft_consensus.cc:3037] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.355120 23586 raft_consensus.cc:504] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.355463 23586 leader_election.cc:288] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [CANDIDATE]: Term 1 election: Requested vote from peers 87045fb026e94528a3f48ff81583ee00 (127.21.112.66:40749), cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.355851 23365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4801be79c3af4653833ea40d2a7fe9c0" candidate_uuid: "1253aad682fc49b9b3475270df2448dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87045fb026e94528a3f48ff81583ee00"
I20260928 23:43:46.355851 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4801be79c3af4653833ea40d2a7fe9c0" candidate_uuid: "1253aad682fc49b9b3475270df2448dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5ebf18bac6425bb22c06567f6e543c"
I20260928 23:43:46.355978 23365 raft_consensus.cc:3037] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.355978 23510 raft_consensus.cc:3037] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.357316 23510 raft_consensus.cc:2454] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1253aad682fc49b9b3475270df2448dc in term 1.
I20260928 23:43:46.357309 23365 raft_consensus.cc:2454] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1253aad682fc49b9b3475270df2448dc in term 1.
I20260928 23:43:46.357756 23161 leader_election.cc:302] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1253aad682fc49b9b3475270df2448dc, 87045fb026e94528a3f48ff81583ee00; no voters:
I20260928 23:43:46.357957 23586 raft_consensus.cc:2781] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:46.358191 23586 raft_consensus.cc:686] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 1 LEADER]: Becoming Leader. State: Replica: 1253aad682fc49b9b3475270df2448dc, State: Running, Role: LEADER
I20260928 23:43:46.358577 23586 consensus_queue.cc:237] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.361985 23071 catalog_manager.cc:5495] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1253aad682fc49b9b3475270df2448dc (127.21.112.65). New cstate: current_term: 1 leader_uuid: "1253aad682fc49b9b3475270df2448dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } health_report { overall_health: UNKNOWN } } }
I20260928 23:43:46.392103 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.388302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.21.112.65:40273)
0928 23:43:46.388437 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.388441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.388455 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:43:46.388596 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.388599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.388653 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.388750 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:46.389023 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.389569 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.390529 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.390759 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.390865 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.391210 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:46.391647 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:46.391744 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.391916 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.391989 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:43:46.398324 23510 raft_consensus.cc:1261] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Refusing update from remote peer 1253aad682fc49b9b3475270df2448dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:46.398763 23365 raft_consensus.cc:1261] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Refusing update from remote peer 1253aad682fc49b9b3475270df2448dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:46.399202 23586 consensus_queue.cc:1035] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:46.399472 23590 consensus_queue.cc:1035] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:46.409126 23595 mvcc.cc:205] Tried to move back new op lower bound from 7334457452120256512 to 7334457451967340544. Current Snapshot: MvccSnapshot[applied={T|T < 7334457452120256512}]
I20260928 23:43:46.409904 23594 mvcc.cc:205] Tried to move back new op lower bound from 7334457452120256512 to 7334457451967340544. Current Snapshot: MvccSnapshot[applied={T|T < 7334457452120256512}]
I20260928 23:43:46.420584 23072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:46.420778 23072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:46.423496 23072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: Sending DeleteTablet for 3 replicas of tablet 4801be79c3af4653833ea40d2a7fe9c0
I20260928 23:43:46.424463 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 4801be79c3af4653833ea40d2a7fe9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51600
I20260928 23:43:46.424471 23347 tablet_service.cc:1514] Processing DeleteTablet for tablet 4801be79c3af4653833ea40d2a7fe9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:44380
I20260928 23:43:46.425072 23604 tablet_replica.cc:330] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: stopping tablet replica
I20260928 23:43:46.425220 23490 tablet_service.cc:1514] Processing DeleteTablet for tablet 4801be79c3af4653833ea40d2a7fe9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:53250
I20260928 23:43:46.425437 23604 raft_consensus.cc:2229] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:46.425623 23072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:43:46.425737 23603 tablet_replica.cc:330] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: stopping tablet replica
I20260928 23:43:46.426038 23603 raft_consensus.cc:2229] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:43:46.426088 23604 raft_consensus.cc:2258] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:43:46.426105 23072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:46.426465 23603 raft_consensus.cc:2258] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:46.427394 23605 tablet_replica.cc:330] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: stopping tablet replica
I20260928 23:43:46.427685 23605 raft_consensus.cc:2229] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:46.427961 23604 ts_tablet_manager.cc:1905] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:46.428005 23605 raft_consensus.cc:2258] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:46.429621 23605 ts_tablet_manager.cc:1905] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:46.430847 23604 ts_tablet_manager.cc:1918] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:43:46.430961 23604 log.cc:1196] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/4801be79c3af4653833ea40d2a7fe9c0
I20260928 23:43:46.431293 23604 ts_tablet_manager.cc:1939] T 4801be79c3af4653833ea40d2a7fe9c0 P 87045fb026e94528a3f48ff81583ee00: Deleting consensus metadata
I20260928 23:43:46.432158 23347 tablet_service.cc:1467] Processing CreateTablet for tablet ceb4c69f5c0243088bee1564294d22fd (DEFAULT_TABLE table=test-table [id=63f3ad8d74c047f989a41786783cf791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:46.432255 23605 ts_tablet_manager.cc:1918] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:43:46.432339 23605 log.cc:1196] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/4801be79c3af4653833ea40d2a7fe9c0
I20260928 23:43:46.432410 23347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb4c69f5c0243088bee1564294d22fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.432641 23605 ts_tablet_manager.cc:1939] T 4801be79c3af4653833ea40d2a7fe9c0 P cb5ebf18bac6425bb22c06567f6e543c: Deleting consensus metadata
I20260928 23:43:46.432752 23057 catalog_manager.cc:4841] TS 87045fb026e94528a3f48ff81583ee00 (127.21.112.66:40749): tablet 4801be79c3af4653833ea40d2a7fe9c0 (table test-table [id=2d6ddf01e16d421998144c6cb1b83dfa]) successfully deleted
I20260928 23:43:46.433759 23057 catalog_manager.cc:4841] TS cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067): tablet 4801be79c3af4653833ea40d2a7fe9c0 (table test-table [id=2d6ddf01e16d421998144c6cb1b83dfa]) successfully deleted
I20260928 23:43:46.434719 23580 tablet_bootstrap.cc:492] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Bootstrap starting.
I20260928 23:43:46.434772 23603 ts_tablet_manager.cc:1905] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:46.434825 23490 tablet_service.cc:1467] Processing CreateTablet for tablet ceb4c69f5c0243088bee1564294d22fd (DEFAULT_TABLE table=test-table [id=63f3ad8d74c047f989a41786783cf791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:46.435066 23490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb4c69f5c0243088bee1564294d22fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.435712 23580 tablet_bootstrap.cc:654] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.437322 23580 tablet_bootstrap.cc:492] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: No bootstrap required, opened a new log
I20260928 23:43:46.437405 23581 tablet_bootstrap.cc:492] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Bootstrap starting.
I20260928 23:43:46.437412 23580 ts_tablet_manager.cc:1397] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:43:46.437848 23603 ts_tablet_manager.cc:1918] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:43:46.437947 23603 log.cc:1196] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/4801be79c3af4653833ea40d2a7fe9c0
I20260928 23:43:46.438006 23580 raft_consensus.cc:348] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.438135 23580 raft_consensus.cc:374] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.438172 23580 raft_consensus.cc:729] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87045fb026e94528a3f48ff81583ee00, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.438297 23603 ts_tablet_manager.cc:1939] T 4801be79c3af4653833ea40d2a7fe9c0 P 1253aad682fc49b9b3475270df2448dc: Deleting consensus metadata
I20260928 23:43:46.438334 23580 consensus_queue.cc:260] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.438627 23580 ts_tablet_manager.cc:1428] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 23:43:46.438727 23581 tablet_bootstrap.cc:654] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.439859 23060 catalog_manager.cc:4841] TS 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273): tablet 4801be79c3af4653833ea40d2a7fe9c0 (table test-table [id=2d6ddf01e16d421998144c6cb1b83dfa]) successfully deleted
I20260928 23:43:46.440207 23210 tablet_service.cc:1467] Processing CreateTablet for tablet ceb4c69f5c0243088bee1564294d22fd (DEFAULT_TABLE table=test-table [id=63f3ad8d74c047f989a41786783cf791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:46.440493 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb4c69f5c0243088bee1564294d22fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.443066 23579 tablet_bootstrap.cc:492] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Bootstrap starting.
I20260928 23:43:46.443374 23581 tablet_bootstrap.cc:492] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: No bootstrap required, opened a new log
I20260928 23:43:46.443480 23581 ts_tablet_manager.cc:1397] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 23:43:46.444037 23581 raft_consensus.cc:348] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.444151 23579 tablet_bootstrap.cc:654] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.444186 23581 raft_consensus.cc:374] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.444227 23581 raft_consensus.cc:729] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5ebf18bac6425bb22c06567f6e543c, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.444368 23581 consensus_queue.cc:260] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.444684 23581 ts_tablet_manager.cc:1428] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:46.445953 23579 tablet_bootstrap.cc:492] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: No bootstrap required, opened a new log
I20260928 23:43:46.446174 23579 ts_tablet_manager.cc:1397] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:43:46.446748 23579 raft_consensus.cc:348] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.446933 23579 raft_consensus.cc:374] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.446982 23579 raft_consensus.cc:729] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1253aad682fc49b9b3475270df2448dc, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.447113 23579 consensus_queue.cc:260] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.447400 23579 ts_tablet_manager.cc:1428] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260928 23:43:46.488897 23417 tablet.cc:2369] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:43:46.489877 23557 tablet.cc:2369] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:43:46.509363 23277 tablet.cc:2369] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:43:46.590524 23585 raft_consensus.cc:482] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:46.590703 23585 raft_consensus.cc:504] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.591881 23585 leader_election.cc:288] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273), cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.596060 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.592292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:56007 (local address 127.21.112.65:40273)
0928 23:43:46.592476 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.592481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.592612 (+ 131us) server_negotiation.cc:407] Connection header received
0928 23:43:46.592726 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.592729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.592783 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.592863 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.593187 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.593919 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.595021 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.595454 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.595582 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.595637 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.595846 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.595923 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:43:46.596072 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.592301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:49583 (local address 127.21.112.67:34067)
0928 23:43:46.592437 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.592442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.592740 (+ 298us) server_negotiation.cc:407] Connection header received
0928 23:43:46.592848 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.592854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.592906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.592984 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:46.593267 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.593837 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.595085 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.595459 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.595588 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.595638 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.595890 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.595959 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:43:46.596086 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.592167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:34067 (local address 127.21.112.66:49583)
0928 23:43:46.592621 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.592636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.592752 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.593020 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.593022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.593037 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.593157 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.593166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.593964 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.593969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.594967 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.594982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.595602 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.595606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.595750 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.595767 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.595860 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.595928 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":39,"thread_start_us":84,"threads_started":1}
I20260928 23:43:46.596096 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.592170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40273 (local address 127.21.112.66:56007)
0928 23:43:46.592519 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.592530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.592630 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.592873 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.592875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.592908 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:46.593036 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.593043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.594036 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.594043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.594874 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.594883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.595581 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.595584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.595719 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.595738 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.595801 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.595889 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":26,"thread_start_us":94,"threads_started":1}
I20260928 23:43:46.596505 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb4c69f5c0243088bee1564294d22fd" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5ebf18bac6425bb22c06567f6e543c" is_pre_election: true
I20260928 23:43:46.596621 23510 raft_consensus.cc:2454] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 0.
I20260928 23:43:46.596948 23230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb4c69f5c0243088bee1564294d22fd" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1253aad682fc49b9b3475270df2448dc" is_pre_election: true
I20260928 23:43:46.597016 23301 leader_election.cc:302] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87045fb026e94528a3f48ff81583ee00, cb5ebf18bac6425bb22c06567f6e543c; no voters:
I20260928 23:43:46.597182 23230 raft_consensus.cc:2454] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 0.
I20260928 23:43:46.597295 23585 raft_consensus.cc:2781] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:46.597361 23585 raft_consensus.cc:482] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:46.597406 23585 raft_consensus.cc:3037] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.598479 23585 raft_consensus.cc:504] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.598843 23585 leader_election.cc:288] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273), cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.599189 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb4c69f5c0243088bee1564294d22fd" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5ebf18bac6425bb22c06567f6e543c"
I20260928 23:43:46.599315 23510 raft_consensus.cc:3037] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.599457 23230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb4c69f5c0243088bee1564294d22fd" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1253aad682fc49b9b3475270df2448dc"
I20260928 23:43:46.599575 23230 raft_consensus.cc:3037] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.600306 23510 raft_consensus.cc:2454] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 1.
I20260928 23:43:46.600587 23301 leader_election.cc:302] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87045fb026e94528a3f48ff81583ee00, cb5ebf18bac6425bb22c06567f6e543c; no voters:
I20260928 23:43:46.600625 23230 raft_consensus.cc:2454] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 1.
I20260928 23:43:46.600907 23585 raft_consensus.cc:2781] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:46.601153 23585 raft_consensus.cc:686] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 1 LEADER]: Becoming Leader. State: Replica: 87045fb026e94528a3f48ff81583ee00, State: Running, Role: LEADER
I20260928 23:43:46.601469 23585 consensus_queue.cc:237] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.604686 23072 catalog_manager.cc:5495] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87045fb026e94528a3f48ff81583ee00 (127.21.112.66). New cstate: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639025, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:46.672607 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.663509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40649 (local address 127.0.0.1:52684)
0928 23:43:46.663875 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.663888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.664047 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.664310 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.664313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.664785 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:46.664901 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.664908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.665733 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.665739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.666330 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.666339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.666842 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.666845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.666927 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.667571 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:46.667583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:46.669095 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:46.670835 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.670847 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.670855 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.671107 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.671399 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:46.671404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:46.671406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:46.671496 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:46.671887 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:46.671895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:46.672014 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.672140 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.672329 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":124,"threads_started":1}
I20260928 23:43:46.672607 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.663543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.21.112.126:40649)
0928 23:43:46.663697 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.663702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.663980 (+ 278us) server_negotiation.cc:407] Connection header received
0928 23:43:46.664150 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.664153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.664199 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.664288 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.665032 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.665625 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.666489 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.666702 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.666819 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.669237 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.669246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.669248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.669268 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.670707 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.671217 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.671222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.671223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.671306 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.671587 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.671592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.671594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.671776 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.671887 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.672191 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.672440 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":51}
I20260928 23:43:46.677040 23072 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:52684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:43:46.677830 23072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:46.683115 23490 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7ad3999814bc3888c700ee2a2a189 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e2e3aa8fa6646988751b4443063ca72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:43:46.683104 23347 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7ad3999814bc3888c700ee2a2a189 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e2e3aa8fa6646988751b4443063ca72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:43:46.683084 23210 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc7ad3999814bc3888c700ee2a2a189 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3e2e3aa8fa6646988751b4443063ca72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:43:46.683501 23490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7ad3999814bc3888c700ee2a2a189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.683501 23347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7ad3999814bc3888c700ee2a2a189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.683501 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc7ad3999814bc3888c700ee2a2a189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:46.686149 23579 tablet_bootstrap.cc:492] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: Bootstrap starting.
I20260928 23:43:46.686914 23581 tablet_bootstrap.cc:492] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: Bootstrap starting.
I20260928 23:43:46.687001 23579 tablet_bootstrap.cc:654] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.687400 23580 tablet_bootstrap.cc:492] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: Bootstrap starting.
I20260928 23:43:46.687904 23581 tablet_bootstrap.cc:654] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.688294 23580 tablet_bootstrap.cc:654] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:46.689184 23581 tablet_bootstrap.cc:492] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: No bootstrap required, opened a new log
I20260928 23:43:46.689275 23581 ts_tablet_manager.cc:1397] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:46.689311 23579 tablet_bootstrap.cc:492] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: No bootstrap required, opened a new log
I20260928 23:43:46.689396 23579 ts_tablet_manager.cc:1397] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 23:43:46.689723 23580 tablet_bootstrap.cc:492] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: No bootstrap required, opened a new log
I20260928 23:43:46.689805 23580 ts_tablet_manager.cc:1397] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:43:46.689788 23581 raft_consensus.cc:348] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.689905 23581 raft_consensus.cc:374] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.689862 23579 raft_consensus.cc:348] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.689942 23581 raft_consensus.cc:729] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5ebf18bac6425bb22c06567f6e543c, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.689966 23579 raft_consensus.cc:374] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.689990 23579 raft_consensus.cc:729] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1253aad682fc49b9b3475270df2448dc, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.690073 23581 consensus_queue.cc:260] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.690114 23579 consensus_queue.cc:260] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.690235 23580 raft_consensus.cc:348] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.690347 23580 raft_consensus.cc:374] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:46.690383 23580 raft_consensus.cc:729] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87045fb026e94528a3f48ff81583ee00, State: Initialized, Role: FOLLOWER
I20260928 23:43:46.690402 23579 ts_tablet_manager.cc:1428] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:43:46.690349 23581 ts_tablet_manager.cc:1428] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:46.690335 23587 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.690500 23587 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.690502 23580 consensus_queue.cc:260] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.690754 23585 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.690889 23585 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.690881 23599 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.691022 23599 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.690769 23580 ts_tablet_manager.cc:1428] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:46.697897 23585 raft_consensus.cc:482] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:46.698007 23585 raft_consensus.cc:504] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.698493 23585 leader_election.cc:288] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273), cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.698957 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7ad3999814bc3888c700ee2a2a189" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5ebf18bac6425bb22c06567f6e543c" is_pre_election: true
I20260928 23:43:46.699102 23510 raft_consensus.cc:2454] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 0.
I20260928 23:43:46.699373 23230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7ad3999814bc3888c700ee2a2a189" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1253aad682fc49b9b3475270df2448dc" is_pre_election: true
I20260928 23:43:46.699414 23301 leader_election.cc:302] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87045fb026e94528a3f48ff81583ee00, cb5ebf18bac6425bb22c06567f6e543c; no voters:
I20260928 23:43:46.699540 23230 raft_consensus.cc:2454] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 0.
I20260928 23:43:46.699584 23585 raft_consensus.cc:2781] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:46.699651 23585 raft_consensus.cc:482] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:46.699713 23585 raft_consensus.cc:3037] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.700722 23585 raft_consensus.cc:504] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.701078 23585 leader_election.cc:288] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273), cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067)
I20260928 23:43:46.701426 23230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7ad3999814bc3888c700ee2a2a189" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1253aad682fc49b9b3475270df2448dc"
I20260928 23:43:46.701428 23510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc7ad3999814bc3888c700ee2a2a189" candidate_uuid: "87045fb026e94528a3f48ff81583ee00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5ebf18bac6425bb22c06567f6e543c"
I20260928 23:43:46.701581 23510 raft_consensus.cc:3037] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.701581 23230 raft_consensus.cc:3037] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:46.702700 23230 raft_consensus.cc:2454] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 1.
I20260928 23:43:46.702713 23510 raft_consensus.cc:2454] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87045fb026e94528a3f48ff81583ee00 in term 1.
I20260928 23:43:46.703063 23301 leader_election.cc:302] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87045fb026e94528a3f48ff81583ee00, cb5ebf18bac6425bb22c06567f6e543c; no voters:
I20260928 23:43:46.703262 23585 raft_consensus.cc:2781] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:46.703377 23585 raft_consensus.cc:686] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [term 1 LEADER]: Becoming Leader. State: Replica: 87045fb026e94528a3f48ff81583ee00, State: Running, Role: LEADER
I20260928 23:43:46.703542 23585 consensus_queue.cc:237] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } }
I20260928 23:43:46.704299 23611 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: TxnStatusTablet state changed. Reason: New leader 87045fb026e94528a3f48ff81583ee00. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.704423 23611 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:43:46.704710 23623 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:43:46.704911 23623 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:43:46.705272 23071 catalog_manager.cc:5495] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87045fb026e94528a3f48ff81583ee00 (127.21.112.66). New cstate: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } health_report { overall_health: UNKNOWN } } }
I20260928 23:43:46.722134 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.717902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.21.112.66:40749)
0928 23:43:46.718033 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.718036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.718050 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:43:46.718224 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.718227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.718279 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.718366 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:46.718686 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.719195 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.720573 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.720775 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.720859 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.721264 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:46.721737 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:46.721872 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.721960 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.722006 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:43:46.722190 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.717796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:40749 (local address 127.0.0.1:44390)
0928 23:43:46.717951 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.717967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.718077 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.718421 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.718424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.718433 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:46.718566 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.718573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.719320 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.719323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.720440 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.720451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.720911 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.720917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.721053 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.721114 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:46.721840 (+ 726us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:46.721858 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.721942 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.722037 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 23:43:46.725087 23510 raft_consensus.cc:1261] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Refusing update from remote peer 87045fb026e94528a3f48ff81583ee00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:46.725252 23230 raft_consensus.cc:1261] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Refusing update from remote peer 87045fb026e94528a3f48ff81583ee00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:46.725620 23611 consensus_queue.cc:1035] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:46.725801 23585 consensus_queue.cc:1035] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:46.727540 23587 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: TxnStatusTablet state changed. Reason: New leader 87045fb026e94528a3f48ff81583ee00. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.727662 23587 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.728142 23599 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: TxnStatusTablet state changed. Reason: New leader 87045fb026e94528a3f48ff81583ee00. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.728251 23599 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.729267 23611 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.729368 23611 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:43:46.732434 23585 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.732542 23585 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 87045fb026e94528a3f48ff81583ee00: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:43:46.732933 23587 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.733058 23587 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P cb5ebf18bac6425bb22c06567f6e543c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.733274 23599 tablet_replica.cc:439] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87045fb026e94528a3f48ff81583ee00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 } } peers { permanent_uuid: "87045fb026e94528a3f48ff81583ee00" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40749 } } peers { permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 } } }
I20260928 23:43:46.733393 23599 tablet_replica.cc:442] T 1fc7ad3999814bc3888c700ee2a2a189 P 1253aad682fc49b9b3475270df2448dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:46.739535 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.735993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.21.112.66:40749)
0928 23:43:46.736143 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.736147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.736159 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:43:46.736199 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.736202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.736252 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.736325 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:46.736637 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.737135 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.738469 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.738693 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.738778 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.739068 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:46.739135 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:46.739196 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.739367 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.739417 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:43:46.751782 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.747640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:46301 (local address 127.21.112.66:40749)
0928 23:43:46.747769 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.747774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.748006 (+ 232us) server_negotiation.cc:407] Connection header received
0928 23:43:46.748145 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.748147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.748190 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.748252 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:46.748567 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.749112 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.750546 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.750830 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.750926 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.751272 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:46.751342 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:46.751396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.751531 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.751574 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:43:46.751816 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.747577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:40749 (local address 127.21.112.66:46301)
0928 23:43:46.747911 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:46.747928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:46.748050 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:46.748292 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:46.748295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:46.748316 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:46.748456 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.748463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.749218 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.749221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.750397 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:46.750410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.750968 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.750972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:46.751075 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.751135 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:46.751434 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:46.751440 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:43:46.751516 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:43:46.751605 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260928 23:43:46.760421 23230 raft_consensus.cc:1261] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Refusing update from remote peer 87045fb026e94528a3f48ff81583ee00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:46.760406 23510 raft_consensus.cc:1261] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Refusing update from remote peer 87045fb026e94528a3f48ff81583ee00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:46.760860 23585 consensus_queue.cc:1035] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1253aad682fc49b9b3475270df2448dc" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:46.761078 23611 consensus_queue.cc:1035] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb5ebf18bac6425bb22c06567f6e543c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:46.778483 23593 mvcc.cc:205] Tried to move back new op lower bound from 7334457453606121472 to 7334457452964827136. Current Snapshot: MvccSnapshot[applied={T|T < 7334457453606121472}]
I20260928 23:43:46.812395 23071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:46.812541 23071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:46.814632 23071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9e0d8c92644348b184f229dc04295f: Sending DeleteTablet for 3 replicas of tablet ceb4c69f5c0243088bee1564294d22fd
I20260928 23:43:46.815176 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb4c69f5c0243088bee1564294d22fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51600
I20260928 23:43:46.815322 23490 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb4c69f5c0243088bee1564294d22fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:53250
I20260928 23:43:46.815412 23603 tablet_replica.cc:330] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: stopping tablet replica
I20260928 23:43:46.815497 23605 tablet_replica.cc:330] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: stopping tablet replica
I20260928 23:43:46.815526 23603 raft_consensus.cc:2229] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:46.815603 23605 raft_consensus.cc:2229] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:46.815649 23603 raft_consensus.cc:2258] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:46.815708 23605 raft_consensus.cc:2258] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:46.816267 23348 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb4c69f5c0243088bee1564294d22fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:44380
I20260928 23:43:46.816426 23604 tablet_replica.cc:330] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: stopping tablet replica
I20260928 23:43:46.816438 23603 ts_tablet_manager.cc:1905] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:46.816517 23604 raft_consensus.cc:2229] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:43:46.816802 23604 raft_consensus.cc:2258] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:46.817520 23604 ts_tablet_manager.cc:1905] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:46.817574 23605 ts_tablet_manager.cc:1905] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:46.819283 23603 ts_tablet_manager.cc:1918] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:46.819353 23603 log.cc:1196] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/ceb4c69f5c0243088bee1564294d22fd
I20260928 23:43:46.819599 23603 ts_tablet_manager.cc:1939] T ceb4c69f5c0243088bee1564294d22fd P 1253aad682fc49b9b3475270df2448dc: Deleting consensus metadata
I20260928 23:43:46.820266 23605 ts_tablet_manager.cc:1918] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:46.820331 23605 log.cc:1196] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/ceb4c69f5c0243088bee1564294d22fd
I20260928 23:43:46.820360 23604 ts_tablet_manager.cc:1918] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:46.820426 23604 log.cc:1196] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/ceb4c69f5c0243088bee1564294d22fd
I20260928 23:43:46.820489 23060 catalog_manager.cc:4841] TS 1253aad682fc49b9b3475270df2448dc (127.21.112.65:40273): tablet ceb4c69f5c0243088bee1564294d22fd (table test-table [id=63f3ad8d74c047f989a41786783cf791]) successfully deleted
I20260928 23:43:46.820578 23605 ts_tablet_manager.cc:1939] T ceb4c69f5c0243088bee1564294d22fd P cb5ebf18bac6425bb22c06567f6e543c: Deleting consensus metadata
I20260928 23:43:46.820698 23604 ts_tablet_manager.cc:1939] T ceb4c69f5c0243088bee1564294d22fd P 87045fb026e94528a3f48ff81583ee00: Deleting consensus metadata
I20260928 23:43:46.821276 23057 catalog_manager.cc:4841] TS 87045fb026e94528a3f48ff81583ee00 (127.21.112.66:40749): tablet ceb4c69f5c0243088bee1564294d22fd (table test-table [id=63f3ad8d74c047f989a41786783cf791]) successfully deleted
I20260928 23:43:46.821430 23057 catalog_manager.cc:4841] TS cb5ebf18bac6425bb22c06567f6e543c (127.21.112.67:34067): tablet ceb4c69f5c0243088bee1564294d22fd (table test-table [id=63f3ad8d74c047f989a41786783cf791]) successfully deleted
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:43:46.833330 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.822187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.21.112.65:40273)
0928 23:43:46.822309 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.822314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.822534 (+ 220us) server_negotiation.cc:407] Connection header received
0928 23:43:46.822633 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.822637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.822684 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.822764 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.823520 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.824312 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.825292 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.825647 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.825756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.829292 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.829308 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.829327 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.829357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.831338 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.831823 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.831830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.831832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.831930 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.832217 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.832220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.832222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.832539 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.832762 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.833027 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.833153 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 23:43:46.833856 23270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51608
I20260928 23:43:46.843528 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:46.835733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.21.112.126:40649)
0928 23:43:46.835866 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:46.835870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:46.835907 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:43:46.836005 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:46.836009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:46.836055 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:46.836139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:46.836778 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.837579 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.838384 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:46.838735 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:46.838849 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:46.840207 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:46.840211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:46.840214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:46.840241 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:46.841786 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.842216 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.842219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.842221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.842313 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:46.842611 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:46.842614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:46.842616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:46.842836 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:46.842966 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:46.843214 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:43:46.843361 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 23:43:46.843972 23071 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52700
I20260928 23:43:46.845610 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23145
I20260928 23:43:46.854671 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23285
I20260928 23:43:46.861123 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23425
I20260928 23:43:46.867134 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23041
2026-09-28T23:43:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2796 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23642](info): set up 2 sockets
Sep 28 23:43:46 dist-test-slave-c3s8 krb5kdc[23642](info): commencing operation
krb5kdc: starting...
W20260928 23:43:47.933898 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:43:48.065819 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46221
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:48.182413 23655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:48.182765 23655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:48.182852 23655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:48.187358 23655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:48.187543 23655 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:43:48.187582 23655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:48.187659 23655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:48.187721 23655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:48.187763 23655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:48.193950 23655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:48.195235 23655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:48.196341 23655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:48.205348 23660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.205382 23663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.205372 23661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:48.205816 23655 server_base.cc:1034] running on GCE node
I20260928 23:43:48.206384 23655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:48.207600 23655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:48.208801 23655 hybrid_clock.cc:648] HybridClock initialized: now 1790639028208787 us; error 36 us; skew 500 ppm
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:48.211817 23655 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:48.212812 23655 webserver.cc:460] Webserver started at http://127.21.112.126:40721/ using document root <none> and password file <none>
I20260928 23:43:48.213394 23655 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:48.213490 23655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:48.213753 23655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:48.215915 23655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "52c1e79867a8336b7426e92766a5d594"
server_key_iv: "3b90314aaf843aab2c763929d3aa205d"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.216490 23655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "52c1e79867a8336b7426e92766a5d594"
server_key_iv: "3b90314aaf843aab2c763929d3aa205d"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.220369 23655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:43:48.222903 23670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:48.224223 23655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:43:48.224375 23655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "52c1e79867a8336b7426e92766a5d594"
server_key_iv: "3b90314aaf843aab2c763929d3aa205d"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.224498 23655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:48.234843 23655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:48.237808 23655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:48.238037 23655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:48.246405 23655 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:46221
I20260928 23:43:48.246436 23732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:46221 every 8 connection(s)
I20260928 23:43:48.247385 23655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:48.250135 23733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:48.252588 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23655
I20260928 23:43:48.252718 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:48.253000 21953 external_mini_cluster.cc:1428] Setting key 78ebcdb24d8219415e0cc30d4c8fffbe
I20260928 23:43:48.256490 23733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Bootstrap starting.
I20260928 23:43:48.258879 23733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:48.259645 23733 log.cc:824] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:48.262277 23733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: No bootstrap required, opened a new log
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:48.265688 23733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46221 } }
I20260928 23:43:48.265959 23733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:48.266053 23733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc3f2ac0f4b441a80ab83482d1e0bde, State: Initialized, Role: FOLLOWER
I20260928 23:43:48.266559 23733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46221 } }
I20260928 23:43:48.266711 23733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:48.266773 23733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:48.266875 23733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:48.267683 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.254469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.21.112.126:46221)
0928 23:43:48.254944 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:48.254959 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:48.254997 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:43:48.255683 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:48.255702 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:48.256049 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:48.256423 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:48.257224 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.258333 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.259288 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.259792 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.260013 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.262898 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:48.262904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:48.262918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:48.262946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:48.264673 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.265167 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.265173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.265178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.265291 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.265603 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.265606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.265608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.266067 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:48.266320 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:48.266560 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 23:43:48.266896 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":137,"threads_started":1}
I20260928 23:43:48.267889 23733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46221 } }
I20260928 23:43:48.268246 23733 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc3f2ac0f4b441a80ab83482d1e0bde; no voters:
I20260928 23:43:48.268628 23733 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:48.268738 23738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:48.268987 23738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [term 1 LEADER]: Becoming Leader. State: Replica: 7bc3f2ac0f4b441a80ab83482d1e0bde, State: Running, Role: LEADER
I20260928 23:43:48.269377 23738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46221 } }
I20260928 23:43:48.269917 23733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:43:48.270675 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc3f2ac0f4b441a80ab83482d1e0bde. Latest consensus state: current_term: 1 leader_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46221 } } }
I20260928 23:43:48.270825 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:48.271209 23747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:48.271461 23741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc3f2ac0f4b441a80ab83482d1e0bde" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46221 } } }
I20260928 23:43:48.271566 23741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:48.274575 23747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:48.280917 23747 catalog_manager.cc:1347] Generated new cluster ID: 3e8abaec252c456ba3ad7b10fccb74f2
I20260928 23:43:48.281024 23747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:48.311570 23747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:48.312574 23747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:48.328776 23747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Generated new TSK 0
I20260928 23:43:48.329725 23747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:43:48.365489 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46221
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:48.480561 23760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:48.480907 23760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:48.480958 23760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:48.484946 23760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:48.485002 23760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:48.485042 23760 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:43:48.485056 23760 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:43:48.485087 23760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:43:48.490455 23760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:48.491607 23760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:48.492571 23760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:48.502416 23768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:48.502503 23760 server_base.cc:1034] running on GCE node
W20260928 23:43:48.502418 23766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.502434 23765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:48.503154 23760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:48.503801 23760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:48.504969 23760 hybrid_clock.cc:648] HybridClock initialized: now 1790639028504953 us; error 29 us; skew 500 ppm
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:48.508126 23760 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:43:48.509074 23760 webserver.cc:460] Webserver started at http://127.21.112.65:41675/ using document root <none> and password file <none>
I20260928 23:43:48.509732 23760 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:48.509799 23760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:48.510043 23760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:48.512238 23760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "ef8052d79de64b1a9e8179df444d7a4d"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "dbb1405330191ef9694ad892dd4c3f3a"
server_key_iv: "0cb17af7af35e535a8427c0c760752a6"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.512810 23760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "ef8052d79de64b1a9e8179df444d7a4d"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "dbb1405330191ef9694ad892dd4c3f3a"
server_key_iv: "0cb17af7af35e535a8427c0c760752a6"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.516763 23760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 23:43:48.519714 23775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:48.521610 23760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:48.521790 23760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "ef8052d79de64b1a9e8179df444d7a4d"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "dbb1405330191ef9694ad892dd4c3f3a"
server_key_iv: "0cb17af7af35e535a8427c0c760752a6"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.521944 23760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:48.552795 23760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:48.556411 23760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:48.556591 23760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:48.558105 23760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:48.558185 23760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:48.558254 23760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:48.558310 23760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:48.569927 23760 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:32875
I20260928 23:43:48.570251 23890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:32875 every 8 connection(s)
I20260928 23:43:48.571148 23760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:43:48.572316 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23760
I20260928 23:43:48.572427 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:43:48.572774 21953 external_mini_cluster.cc:1428] Setting key f19b6a791a3334d34360f2b8f7661510
I20260928 23:43:48.573139 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.559133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:33857 (local address 127.21.112.126:46221)
0928 23:43:48.559283 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:48.559286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:48.560470 (+ 1184us) server_negotiation.cc:407] Connection header received
0928 23:43:48.561794 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:48.561798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:48.561859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:48.561966 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:48.563364 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.563979 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.565413 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.565791 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.565912 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.568899 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:48.568902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:48.568904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:48.568929 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:48.570748 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.571360 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.571363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.571364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.571439 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.571850 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.571857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.571860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.572147 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:48.572266 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:48.572840 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 23:43:48.572980 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:43:48.574054 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.559715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.21.112.65:33857)
0928 23:43:48.560301 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:48.560355 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:48.561484 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:48.562164 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:48.562172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:48.562586 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:48.563194 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.563214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.564106 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.564110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.565293 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.565302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.565958 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.565962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.566138 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.566758 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:48.566790 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:48.568705 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:48.570866 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.570872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.570887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.571269 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.571536 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.571539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.571541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.571725 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.572262 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:48.572274 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:48.572577 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 23:43:48.572754 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 23:43:48.573066 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":201,"threads_started":1}
I20260928 23:43:48.575294 23891 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46221
I20260928 23:43:48.575500 23891 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:48.576129 23891 heartbeater.cc:511] Master 127.21.112.126:46221 requested a full tablet report, sending...
I20260928 23:43:48.577898 23686 ts_manager.cc:194] Registered new tserver with Master: ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875)
I20260928 23:43:48.579365 23686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:33857
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:43:48.586295 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.577793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:48395 (local address 127.21.112.126:46221)
0928 23:43:48.577929 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:48.577935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:48.578165 (+ 230us) server_negotiation.cc:407] Connection header received
0928 23:43:48.578261 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:48.578267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:48.578324 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:48.578410 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:48.578901 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.579765 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.581197 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.581613 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.581691 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.583108 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:48.583111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:48.583113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:48.583134 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:48.584585 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.585039 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.585043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.585045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.585125 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.585413 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.585417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.585419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.585665 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:48.585765 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:48.586026 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:43:48.586134 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:43:48.586305 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.577765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.21.112.65:48395)
0928 23:43:48.578072 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:48.578087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:48.578183 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:48.578446 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:48.578449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:48.578655 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:48.578775 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.578781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.579941 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.579948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.581053 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.581068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.581870 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.581873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.581986 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.582487 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:48.582501 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:48.582967 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:48.584718 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.584722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.584724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.584939 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.585225 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.585228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.585230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.585316 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.585766 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:48.585768 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:48.585867 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 23:43:48.585975 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 23:43:48.586121 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:43:48.605278 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46221
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:48.719460 23900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:48.719786 23900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:48.719871 23900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:48.723923 23900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:48.724035 23900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:48.724112 23900 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:43:48.724156 23900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:43:48.724220 23900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:43:48.729946 23900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:48.731114 23900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:48.732193 23900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:48.742115 23906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.742184 23905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.742178 23908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:48.742426 23900 server_base.cc:1034] running on GCE node
I20260928 23:43:48.742802 23900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:48.743382 23900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:48.744556 23900 hybrid_clock.cc:648] HybridClock initialized: now 1790639028744518 us; error 51 us; skew 500 ppm
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:48.747573 23900 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:43:48.748468 23900 webserver.cc:460] Webserver started at http://127.21.112.66:44111/ using document root <none> and password file <none>
I20260928 23:43:48.749049 23900 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:48.749114 23900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:48.749284 23900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:48.751446 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "357076be62ee53113152dcb98b616a39"
server_key_iv: "4075db58f1cbcd82adbe64671db01964"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.752007 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "357076be62ee53113152dcb98b616a39"
server_key_iv: "4075db58f1cbcd82adbe64671db01964"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.755998 23900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:43:48.758586 23915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:48.759931 23900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:48.760064 23900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "357076be62ee53113152dcb98b616a39"
server_key_iv: "4075db58f1cbcd82adbe64671db01964"
server_key_version: "encryptionkey@0"
I20260928 23:43:48.760182 23900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:48.798714 23900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:48.802335 23900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:48.802536 23900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:48.804059 23900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:48.804111 23900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:48.804174 23900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:48.804209 23900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:48.815220 23900 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:45333
I20260928 23:43:48.815948 24030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:45333 every 8 connection(s)
I20260928 23:43:48.816397 23900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:43:48.819584 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.805169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:37515 (local address 127.21.112.126:46221)
0928 23:43:48.805339 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:48.805345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:48.806344 (+ 999us) server_negotiation.cc:407] Connection header received
0928 23:43:48.807864 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:48.807868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:48.807927 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:48.808014 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:48.809763 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.810373 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.812206 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.812507 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.812590 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.815904 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:48.815908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:48.815910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:48.815938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:48.817685 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.818152 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.818154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.818156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.818231 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.818522 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.818524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.818525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.818794 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:48.818876 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:48.819360 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 23:43:48.819464 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:43:48.820519 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.805662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.21.112.66:37515)
0928 23:43:48.806175 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:48.806239 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:48.807602 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:48.808220 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:48.808231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:48.808774 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:48.809576 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.809602 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.810521 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.810526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.812067 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.812079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.812662 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.812670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.812922 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.813771 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:48.813798 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:48.815695 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:48.817780 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.817785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.817795 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.818056 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.818289 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.818291 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.818293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.818419 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.818892 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:48.818898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:48.819135 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 23:43:48.819297 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:43:48.819590 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260928 23:43:48.821904 24031 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46221
I20260928 23:43:48.822065 24031 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:48.822613 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 23900
I20260928 23:43:48.822687 24031 heartbeater.cc:511] Master 127.21.112.126:46221 requested a full tablet report, sending...
I20260928 23:43:48.822723 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:43:48.823060 21953 external_mini_cluster.cc:1428] Setting key 1f5a5c9448c4793b1b78f693a14b4013
I20260928 23:43:48.823913 23687 ts_manager.cc:194] Registered new tserver with Master: e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333)
I20260928 23:43:48.824699 23687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:37515
I20260928 23:43:48.832973 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.824439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.21.112.66:57477)
0928 23:43:48.824724 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:48.824739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:48.824882 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:48.825171 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:48.825174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:48.825443 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:48.825587 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.825595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.826704 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.826711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.827361 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:48.827370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.828005 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.828008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:48.828103 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.828716 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:48.828731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:48.829147 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:48.831323 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.831326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.831327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.831526 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.831894 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:48.831897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:48.831898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:48.831982 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:48.832513 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:48.832515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:48.832594 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 23:43:48.832682 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:43:48.832828 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260928 23:43:48.833092 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:48.824313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:57477 (local address 127.21.112.126:46221)
0928 23:43:48.824447 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:48.824452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:48.824816 (+ 364us) server_negotiation.cc:407] Connection header received
0928 23:43:48.824998 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:48.825001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:48.825039 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:48.825144 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:48.825726 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.826557 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.827496 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:48.827874 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:48.827999 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:48.829328 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:48.829335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:48.829340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:48.829373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:48.831212 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.831633 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.831638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.831640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.831737 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:48.832091 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:48.832095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:48.832097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:48.832413 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:48.832526 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:48.832799 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:43:48.832931 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:43:48.853984 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46221
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:48.967523 24040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:48.967809 24040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:48.967905 24040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:48.972235 24040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:48.972296 24040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:48.972368 24040 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:43:48.972399 24040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:43:48.972476 24040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:43:48.978060 24040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:46221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:48.979162 24040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:48.980274 24040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:48.990244 24048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.990309 24046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:48.990244 24045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:48.990455 24040 server_base.cc:1034] running on GCE node
I20260928 23:43:48.991225 24040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:48.991819 24040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:48.992982 24040 hybrid_clock.cc:648] HybridClock initialized: now 1790639028992947 us; error 48 us; skew 500 ppm
Sep 28 23:43:48 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:48.995755 24040 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:43:48.996620 24040 webserver.cc:460] Webserver started at http://127.21.112.67:34329/ using document root <none> and password file <none>
I20260928 23:43:48.997154 24040 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:48.997211 24040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:48.997385 24040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:48.999462 24040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "ab0ba9074da6430d89112f9cff7e54b4"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "3817d3030dfccb7811cf3d9a9f8857a4"
server_key_iv: "747f7636c2aa2ea7625e10f6926242f8"
server_key_version: "encryptionkey@0"
I20260928 23:43:49.000056 24040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "ab0ba9074da6430d89112f9cff7e54b4"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "3817d3030dfccb7811cf3d9a9f8857a4"
server_key_iv: "747f7636c2aa2ea7625e10f6926242f8"
server_key_version: "encryptionkey@0"
I20260928 23:43:49.003808 24040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 23:43:49.006315 24055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:49.007614 24040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:49.007741 24040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "ab0ba9074da6430d89112f9cff7e54b4"
format_stamp: "Formatted at 2026-09-28 23:43:48 on dist-test-slave-c3s8"
server_key: "3817d3030dfccb7811cf3d9a9f8857a4"
server_key_iv: "747f7636c2aa2ea7625e10f6926242f8"
server_key_version: "encryptionkey@0"
I20260928 23:43:49.007861 24040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:49.018273 24040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:49.021896 24040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:49.022105 24040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:49.023660 24040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:49.023711 24040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:49.023754 24040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:49.023808 24040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:43:49 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:49.035071 24040 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:36521
I20260928 23:43:49.035131 24170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:36521 every 8 connection(s)
I20260928 23:43:49.036248 24040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:43:49.040150 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.024673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:34405 (local address 127.21.112.126:46221)
0928 23:43:49.024946 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.024952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.025829 (+ 877us) server_negotiation.cc:407] Connection header received
0928 23:43:49.027139 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.027143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.027236 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.027344 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:49.029113 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.030056 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.031929 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.032303 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.032437 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.035668 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:49.035673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:49.035675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:49.035709 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:49.037624 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.038388 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.038391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.038392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.038483 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.038960 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.038963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.038964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.039211 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:49.039320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.039863 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.040022 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 23:43:49.040616 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24040
I20260928 23:43:49.040751 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:43:49.041044 21953 external_mini_cluster.cc:1428] Setting key 123df92927d6e1523be517b0b5a27d8e
I20260928 23:43:49.041172 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.025155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.21.112.67:34405)
0928 23:43:49.025664 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.025726 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.026893 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.027601 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.027612 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.028195 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:49.028954 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.028977 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.030214 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.030217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.031800 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.031811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.032464 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.032466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.032620 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.033433 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:49.033454 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:49.035477 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:49.037745 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:49.037751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:49.037768 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:49.038207 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:49.038611 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:49.038614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:49.038615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:49.038757 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:49.039353 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:49.039358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:49.039608 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.039784 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.040071 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260928 23:43:49.042511 24171 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46221
I20260928 23:43:49.042703 24171 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:49.043351 24171 heartbeater.cc:511] Master 127.21.112.126:46221 requested a full tablet report, sending...
I20260928 23:43:49.044561 23687 ts_manager.cc:194] Registered new tserver with Master: ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:49.045239 23687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:34405
I20260928 23:43:49.052376 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.044658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.21.112.67:44051)
0928 23:43:49.045037 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.045049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.045179 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.045478 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.045480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.045719 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:49.045860 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.045869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.046615 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.046618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.047421 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.047429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.047916 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.047919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.048001 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.048492 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:49.048504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:49.048831 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:49.050801 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:49.050803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:49.050805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:49.051036 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:49.051337 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:49.051340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:49.051342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:49.051435 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:49.051881 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:49.051884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:49.051978 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.052079 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.052208 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
I20260928 23:43:49.052423 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.044763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:44051 (local address 127.21.112.126:46221)
0928 23:43:49.044901 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.044905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.045128 (+ 223us) server_negotiation.cc:407] Connection header received
0928 23:43:49.045278 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.045282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.045360 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.045461 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:49.045993 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.046488 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.047532 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.047745 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.047827 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.049009 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:49.049012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:49.049014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:49.049037 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:49.050687 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.051162 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.051164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.051166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.051247 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.051518 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.051520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.051521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.051702 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:49.051864 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.052188 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.052306 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:43:49.055485 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:43:49 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:43:49 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:49.082861 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.074081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.21.112.126:46221)
0928 23:43:49.074245 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.074250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.074264 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:43:49.074310 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.074312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.074360 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.074486 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:49.075237 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.075895 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.076808 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.077033 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.077144 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.079636 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:49.079639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:49.079641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:49.079671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:49.081183 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.081641 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.081643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.081645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.081721 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.081997 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.081999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.082001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.082172 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:49.082268 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.082533 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.082683 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:43:49.085937 23687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:43:49.089042 23687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:49.105043 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.099277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.21.112.67:36521)
0928 23:43:49.099737 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.099750 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.099773 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:43:49.099848 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.099853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.100105 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.100333 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.100914 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.102175 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.103730 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.104342 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.104475 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.104657 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.104783 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.104886 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":103,"threads_started":1}
I20260928 23:43:49.105047 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.098603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:36521 (local address 127.0.0.1:40206)
0928 23:43:49.099464 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.099479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.099628 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.100502 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.100510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.100535 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.100779 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.100790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.102303 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.102306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.103599 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.103613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.104476 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.104480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.104592 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.104629 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.104752 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.104883 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":151,"spinlock_wait_cycles":8448,"thread_start_us":85,"threads_started":1}
I20260928 23:43:49.105453 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.099224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:32875 (local address 127.0.0.1:38058)
0928 23:43:49.100021 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.100033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.100113 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.100684 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.100686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.100700 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.100867 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.100881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.102353 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.102356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.103757 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.103765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.104886 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.104890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.105086 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.105120 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.105199 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.105286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":367,"thread_start_us":97,"threads_started":1}
I20260928 23:43:49.105662 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.098386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45333 (local address 127.0.0.1:37764)
0928 23:43:49.099331 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.099393 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.099629 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.100601 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.100604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.100616 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.100823 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.100833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.101905 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.101908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.103494 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.103505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.104886 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.104890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.105082 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.105280 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.105354 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.105419 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":445,"spinlock_wait_cycles":3840,"thread_start_us":101,"threads_started":1}
I20260928 23:43:49.105881 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.099699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.21.112.65:32875)
0928 23:43:49.100084 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.100090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.100113 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:43:49.100256 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.100261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.100472 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.100618 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.101063 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.102176 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.103870 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.104628 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.104716 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.104828 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.105629 (+ 801us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.105732 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":107,"threads_started":1}
I20260928 23:43:49.106531 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.098542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.21.112.66:45333)
0928 23:43:49.099224 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.099236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.099492 (+ 256us) server_negotiation.cc:407] Connection header received
0928 23:43:49.099854 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.099860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.100467 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.100618 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.100937 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.101788 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.103633 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.104529 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.104641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.104818 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.106282 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.106398 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":146,"threads_started":1}
I20260928 23:43:49.108633 23965 tablet_service.cc:1467] Processing CreateTablet for tablet 423551a9615848e6bf1c57ca3950016c (DEFAULT_TABLE table=test-table [id=6bde009fca584e15962ca6d031e3cf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:49.108502 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 423551a9615848e6bf1c57ca3950016c (DEFAULT_TABLE table=test-table [id=6bde009fca584e15962ca6d031e3cf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:49.108502 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 423551a9615848e6bf1c57ca3950016c (DEFAULT_TABLE table=test-table [id=6bde009fca584e15962ca6d031e3cf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:49.109643 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 423551a9615848e6bf1c57ca3950016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.109644 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 423551a9615848e6bf1c57ca3950016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.109644 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 423551a9615848e6bf1c57ca3950016c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.116544 24194 tablet_bootstrap.cc:492] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Bootstrap starting.
I20260928 23:43:49.117834 24196 tablet_bootstrap.cc:492] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Bootstrap starting.
I20260928 23:43:49.119107 24194 tablet_bootstrap.cc:654] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.119675 24196 tablet_bootstrap.cc:654] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.119987 24194 log.cc:824] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:49.120409 24196 log.cc:824] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:49.120687 24195 tablet_bootstrap.cc:492] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Bootstrap starting.
I20260928 23:43:49.122447 24196 tablet_bootstrap.cc:492] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: No bootstrap required, opened a new log
I20260928 23:43:49.122677 24196 ts_tablet_manager.cc:1397] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 23:43:49.122862 24195 tablet_bootstrap.cc:654] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.123034 24194 tablet_bootstrap.cc:492] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: No bootstrap required, opened a new log
I20260928 23:43:49.123219 24194 ts_tablet_manager.cc:1397] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:43:49.123664 24195 log.cc:824] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:49.126282 24196 raft_consensus.cc:348] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.126528 24196 raft_consensus.cc:374] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.126595 24196 raft_consensus.cc:729] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0ba9074da6430d89112f9cff7e54b4, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.126439 24194 raft_consensus.cc:348] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.126708 24194 raft_consensus.cc:374] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.126783 24194 raft_consensus.cc:729] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.127110 24196 consensus_queue.cc:260] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.127220 24194 consensus_queue.cc:260] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.127456 24195 tablet_bootstrap.cc:492] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: No bootstrap required, opened a new log
I20260928 23:43:49.127645 24195 ts_tablet_manager.cc:1397] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:43:49.127810 24171 heartbeater.cc:503] Master 127.21.112.126:46221 was elected leader, sending a full tablet report...
I20260928 23:43:49.128273 24194 ts_tablet_manager.cc:1428] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:43:49.128630 24031 heartbeater.cc:503] Master 127.21.112.126:46221 was elected leader, sending a full tablet report...
I20260928 23:43:49.129429 24196 ts_tablet_manager.cc:1428] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:43:49.132273 24195 raft_consensus.cc:348] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.132524 24195 raft_consensus.cc:374] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.132604 24195 raft_consensus.cc:729] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8052d79de64b1a9e8179df444d7a4d, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.133096 24195 consensus_queue.cc:260] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.133756 23891 heartbeater.cc:503] Master 127.21.112.126:46221 was elected leader, sending a full tablet report...
I20260928 23:43:49.134049 24195 ts_tablet_manager.cc:1428] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:43:49.236729 24201 raft_consensus.cc:482] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:49.236928 24201 raft_consensus.cc:504] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.238233 24201 leader_election.cc:288] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:49.241971 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.238447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:32875 (local address 127.21.112.66:58021)
0928 23:43:49.238568 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.238587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.238689 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.239026 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.239029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.239047 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.239180 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.239185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.239932 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.239937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.240755 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.240763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.241433 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.241437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.241608 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.241627 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.241716 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.241788 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":6,"spinlock_wait_cycles":4864}
I20260928 23:43:49.241961 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.238548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:58021 (local address 127.21.112.65:32875)
0928 23:43:49.238690 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.238696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.238715 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:43:49.238778 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.238781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.238896 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.239009 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.239302 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.239808 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.240893 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.241304 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.241433 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.241487 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.241734 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.241809 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:43:49.242648 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.238531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:36521 (local address 127.21.112.66:47731)
0928 23:43:49.238950 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.238961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.239072 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.239324 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.239327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.239341 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.239824 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.239830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.240540 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.240546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.241468 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.241477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.242222 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.242226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.242324 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.242351 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.242444 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.242507 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":53,"thread_start_us":142,"threads_started":1}
I20260928 23:43:49.242647 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.238755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:47731 (local address 127.21.112.67:36521)
0928 23:43:49.238873 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.238877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.239036 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:43:49.239184 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.239192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.239230 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.239315 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.239944 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.240432 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.241637 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.242102 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.242222 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.242267 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.242457 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.242522 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:43:49.242731 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423551a9615848e6bf1c57ca3950016c" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d" is_pre_election: true
I20260928 23:43:49.243119 23845 raft_consensus.cc:2454] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 0.
I20260928 23:43:49.243454 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423551a9615848e6bf1c57ca3950016c" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4" is_pre_election: true
I20260928 23:43:49.243669 23919 leader_election.cc:302] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e30ae3be03d04f17a3dd6ba8b18beb2c, ef8052d79de64b1a9e8179df444d7a4d; no voters:
I20260928 23:43:49.243734 24125 raft_consensus.cc:2454] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 0.
I20260928 23:43:49.243976 24201 raft_consensus.cc:2781] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:49.244081 24201 raft_consensus.cc:482] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:49.244129 24201 raft_consensus.cc:3037] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:49.245190 24201 raft_consensus.cc:504] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.245522 24201 leader_election.cc:288] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 election: Requested vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:49.245862 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423551a9615848e6bf1c57ca3950016c" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4"
I20260928 23:43:49.245887 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "423551a9615848e6bf1c57ca3950016c" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d"
I20260928 23:43:49.245995 24125 raft_consensus.cc:3037] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:49.246065 23845 raft_consensus.cc:3037] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:49.247272 24125 raft_consensus.cc:2454] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 1.
I20260928 23:43:49.247361 23845 raft_consensus.cc:2454] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 1.
I20260928 23:43:49.247584 23918 leader_election.cc:302] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, e30ae3be03d04f17a3dd6ba8b18beb2c; no voters:
I20260928 23:43:49.247784 24201 raft_consensus.cc:2781] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:49.247997 24201 raft_consensus.cc:686] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 LEADER]: Becoming Leader. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Running, Role: LEADER
I20260928 23:43:49.248338 24201 consensus_queue.cc:237] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.251557 23686 catalog_manager.cc:5495] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c reported cstate change: term changed from 0 to 1, leader changed from <none> to e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66). New cstate: current_term: 1 leader_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } health_report { overall_health: UNKNOWN } } }
W20260928 23:43:49.288347 24172 tablet.cc:2369] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 23:43:49 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639028, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
W20260928 23:43:49.317905 24032 tablet.cc:2369] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:43:49.320725 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.311519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.21.112.126:46221)
0928 23:43:49.311650 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.311654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.312043 (+ 389us) server_negotiation.cc:407] Connection header received
0928 23:43:49.312212 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.312216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.312277 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.312371 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:49.313191 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.313892 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.314703 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.314926 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.315038 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.317416 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:49.317420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:49.317422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:49.317442 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:49.318949 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.319420 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.319425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.319427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.319501 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:49.319770 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:49.319774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:49.319778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:49.319994 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:49.320099 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.320398 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.320536 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:43:49.320755 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.311518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46221 (local address 127.0.0.1:44196)
0928 23:43:49.311950 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.311964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.312110 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.312389 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.312391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.312972 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:49.313071 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.313078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.314009 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.314012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.314581 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.314588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.315043 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.315047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.315134 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.315705 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:49.315717 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:49.317263 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:49.319062 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:49.319074 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:49.319081 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:49.319320 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:49.319593 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:49.319597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:49.319599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:49.319684 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:49.320110 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:49.320118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:49.320234 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.320353 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.320605 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":99,"thread_start_us":157,"threads_started":1}
W20260928 23:43:49.322855 23892 tablet.cc:2369] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:43:49.325147 23686 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:44196:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:43:49.325878 23686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:49.331205 23965 tablet_service.cc:1467] Processing CreateTablet for tablet afaf1c9ce7cb437399de3b2c66fedcab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44c282d0e95c4e748b0cf585f33c594c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:43:49.331205 23825 tablet_service.cc:1467] Processing CreateTablet for tablet afaf1c9ce7cb437399de3b2c66fedcab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44c282d0e95c4e748b0cf585f33c594c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:43:49.331437 24105 tablet_service.cc:1467] Processing CreateTablet for tablet afaf1c9ce7cb437399de3b2c66fedcab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=44c282d0e95c4e748b0cf585f33c594c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:43:49.332020 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afaf1c9ce7cb437399de3b2c66fedcab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.332020 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afaf1c9ce7cb437399de3b2c66fedcab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.332041 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afaf1c9ce7cb437399de3b2c66fedcab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.334703 24196 tablet_bootstrap.cc:492] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: Bootstrap starting.
I20260928 23:43:49.334932 24194 tablet_bootstrap.cc:492] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: Bootstrap starting.
I20260928 23:43:49.335765 24196 tablet_bootstrap.cc:654] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.335848 24194 tablet_bootstrap.cc:654] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.336326 24195 tablet_bootstrap.cc:492] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: Bootstrap starting.
I20260928 23:43:49.337007 24194 tablet_bootstrap.cc:492] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: No bootstrap required, opened a new log
I20260928 23:43:49.337064 24194 ts_tablet_manager.cc:1397] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:49.337114 24195 tablet_bootstrap.cc:654] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.337467 24194 raft_consensus.cc:348] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.337651 24194 raft_consensus.cc:374] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.337693 24194 raft_consensus.cc:729] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.337826 24194 consensus_queue.cc:260] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.338112 24194 ts_tablet_manager.cc:1428] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:49.338073 24201 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.338187 24201 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.338208 24196 tablet_bootstrap.cc:492] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: No bootstrap required, opened a new log
I20260928 23:43:49.338270 24196 ts_tablet_manager.cc:1397] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:43:49.338677 24196 raft_consensus.cc:348] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.338796 24196 raft_consensus.cc:374] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.338840 24196 raft_consensus.cc:729] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0ba9074da6430d89112f9cff7e54b4, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.338970 24196 consensus_queue.cc:260] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.339207 24196 ts_tablet_manager.cc:1428] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:43:49.339193 24200 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.339355 24200 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.340090 24195 tablet_bootstrap.cc:492] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: No bootstrap required, opened a new log
I20260928 23:43:49.340186 24195 ts_tablet_manager.cc:1397] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 23:43:49.340739 24195 raft_consensus.cc:348] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.340869 24195 raft_consensus.cc:374] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.340915 24195 raft_consensus.cc:729] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8052d79de64b1a9e8179df444d7a4d, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.341042 24195 consensus_queue.cc:260] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.341305 24195 ts_tablet_manager.cc:1428] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:43:49.341320 24202 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.341475 24202 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.530932 24200 raft_consensus.cc:482] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:49.531077 24200 raft_consensus.cc:504] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.532253 24200 leader_election.cc:288] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333)
I20260928 23:43:49.536926 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.532675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:48737 (local address 127.21.112.65:32875)
0928 23:43:49.532897 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.532902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.532924 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:43:49.532975 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.532977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.533015 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.533117 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.533446 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.534276 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.535408 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.536161 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.536324 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.536412 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.536665 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.536783 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260928 23:43:49.536959 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.532544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:32875 (local address 127.21.112.67:48737)
0928 23:43:49.532665 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.532684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.532813 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.533130 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.533133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.533151 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.533299 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.533307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.534392 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.534399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.535265 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.535274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.536304 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.536308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.536471 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.536528 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.536637 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.536791 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:43:49.536950 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.533379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:35245 (local address 127.21.112.66:45333)
0928 23:43:49.533496 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.533500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.533511 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:43:49.533600 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.533603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.533650 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.533710 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:49.534031 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.534525 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.535596 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.536305 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.536423 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.536472 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.536776 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.536845 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:43:49.537281 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.532932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45333 (local address 127.21.112.67:35245)
0928 23:43:49.533236 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.533262 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.533357 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.533751 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.533754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.533775 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:49.533908 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.533915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.534639 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.534645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.535467 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.535476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.536538 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.536541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.536648 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.536667 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.537075 (+ 408us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.537151 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260928 23:43:49.537623 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afaf1c9ce7cb437399de3b2c66fedcab" candidate_uuid: "ab0ba9074da6430d89112f9cff7e54b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d" is_pre_election: true
I20260928 23:43:49.537796 23845 raft_consensus.cc:2454] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab0ba9074da6430d89112f9cff7e54b4 in term 0.
I20260928 23:43:49.538074 23985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afaf1c9ce7cb437399de3b2c66fedcab" candidate_uuid: "ab0ba9074da6430d89112f9cff7e54b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" is_pre_election: true
I20260928 23:43:49.538170 24059 leader_election.cc:302] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, ef8052d79de64b1a9e8179df444d7a4d; no voters:
I20260928 23:43:49.538292 23985 raft_consensus.cc:2454] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab0ba9074da6430d89112f9cff7e54b4 in term 0.
I20260928 23:43:49.538426 24200 raft_consensus.cc:2781] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:49.538486 24200 raft_consensus.cc:482] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:49.538513 24200 raft_consensus.cc:3037] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:49.539628 24200 raft_consensus.cc:504] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.539973 24200 leader_election.cc:288] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [CANDIDATE]: Term 1 election: Requested vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333)
I20260928 23:43:49.540390 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afaf1c9ce7cb437399de3b2c66fedcab" candidate_uuid: "ab0ba9074da6430d89112f9cff7e54b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d"
I20260928 23:43:49.540429 23985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afaf1c9ce7cb437399de3b2c66fedcab" candidate_uuid: "ab0ba9074da6430d89112f9cff7e54b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c"
I20260928 23:43:49.540494 23845 raft_consensus.cc:3037] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:49.540604 23985 raft_consensus.cc:3037] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:49.541440 23845 raft_consensus.cc:2454] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab0ba9074da6430d89112f9cff7e54b4 in term 1.
I20260928 23:43:49.541697 23985 raft_consensus.cc:2454] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab0ba9074da6430d89112f9cff7e54b4 in term 1.
I20260928 23:43:49.542062 24059 leader_election.cc:302] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, e30ae3be03d04f17a3dd6ba8b18beb2c; no voters:
I20260928 23:43:49.542289 24200 raft_consensus.cc:2781] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:49.542565 24200 raft_consensus.cc:686] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 LEADER]: Becoming Leader. State: Replica: ab0ba9074da6430d89112f9cff7e54b4, State: Running, Role: LEADER
I20260928 23:43:49.542932 24200 consensus_queue.cc:237] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.545024 24216 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: TxnStatusTablet state changed. Reason: New leader ab0ba9074da6430d89112f9cff7e54b4. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.545166 24216 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:43:49.545516 24218 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:43:49.545802 24218 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:43:49.546928 23687 catalog_manager.cc:5495] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } health_report { overall_health: HEALTHY } } }
I20260928 23:43:49.582587 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.579053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.21.112.67:36521)
0928 23:43:49.579194 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.579201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.579216 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:43:49.579290 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.579293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.579344 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.579452 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:49.579710 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.580253 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.581122 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.581347 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.581467 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.581778 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.582183 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.582319 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.582397 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.582448 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:43:49.582585 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.578806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:36521 (local address 127.0.0.1:40208)
0928 23:43:49.578977 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.578994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.579096 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.579465 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.579468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.579476 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:49.579584 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.579591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.580380 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.580384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.580996 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.581004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.581459 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.581463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.581577 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.581631 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.582292 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.582308 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.582387 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.582470 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 23:43:49.587687 23985 raft_consensus.cc:1261] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Refusing update from remote peer ab0ba9074da6430d89112f9cff7e54b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:49.587702 23845 raft_consensus.cc:1261] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Refusing update from remote peer ab0ba9074da6430d89112f9cff7e54b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:49.588346 24200 consensus_queue.cc:1035] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:49.588537 24216 consensus_queue.cc:1035] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:49.591984 24201 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: TxnStatusTablet state changed. Reason: New leader ab0ba9074da6430d89112f9cff7e54b4. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.592180 24201 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.593199 24202 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: TxnStatusTablet state changed. Reason: New leader ab0ba9074da6430d89112f9cff7e54b4. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.593341 24202 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.596088 24216 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.596235 24216 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:43:49.596295 24202 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.596362 24216 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.596431 24216 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ab0ba9074da6430d89112f9cff7e54b4: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:43:49.596427 24202 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P ef8052d79de64b1a9e8179df444d7a4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.596531 24221 mvcc.cc:205] Tried to move back new op lower bound from 7334457465184169984 to 7334457465010937856. Current Snapshot: MvccSnapshot[applied={T|T < 7334457465184169984}]
I20260928 23:43:49.597059 24223 mvcc.cc:205] Tried to move back new op lower bound from 7334457465184169984 to 7334457465010937856. Current Snapshot: MvccSnapshot[applied={T|T < 7334457465184169984}]
I20260928 23:43:49.598623 24201 tablet_replica.cc:439] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab0ba9074da6430d89112f9cff7e54b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } } }
I20260928 23:43:49.598731 24201 tablet_replica.cc:442] T afaf1c9ce7cb437399de3b2c66fedcab P e30ae3be03d04f17a3dd6ba8b18beb2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:43:49.607187 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.602641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.21.112.66:45333)
0928 23:43:49.602751 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.602755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.602990 (+ 235us) server_negotiation.cc:407] Connection header received
0928 23:43:49.603142 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.603144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.603226 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.603340 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:49.603611 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.604110 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.605552 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.605859 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.606003 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.606319 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.606779 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.606871 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.606990 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.607040 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:43:49.620532 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.616452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:57883 (local address 127.21.112.67:36521)
0928 23:43:49.616595 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.616599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.616718 (+ 119us) server_negotiation.cc:407] Connection header received
0928 23:43:49.616862 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.616864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.616909 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.616975 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:49.617301 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.617883 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.619308 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.619557 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.619657 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.620057 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.620137 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.620202 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.620375 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.620426 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:43:49.620602 24233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.616251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:36521 (local address 127.21.112.66:57883)
0928 23:43:49.616625 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.616641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.616761 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.617013 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.617016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.617025 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:49.617187 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.617194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.617999 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.618003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.619106 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.619118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.619687 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.619692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.619811 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.619872 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.620235 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.620242 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.620330 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.620415 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":150,"threads_started":1}
I20260928 23:43:49.629228 23845 raft_consensus.cc:1261] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Refusing update from remote peer e30ae3be03d04f17a3dd6ba8b18beb2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:49.629477 24125 raft_consensus.cc:1261] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Refusing update from remote peer e30ae3be03d04f17a3dd6ba8b18beb2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:49.629796 24204 consensus_queue.cc:1035] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:49.630059 24201 consensus_queue.cc:1035] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:49.635157 24222 mvcc.cc:205] Tried to move back new op lower bound from 7334457465357258752 to 7334457463803731968. Current Snapshot: MvccSnapshot[applied={T|T < 7334457465357258752}]
I20260928 23:43:49.655961 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.651676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45333 (local address 127.21.112.67:50409)
0928 23:43:49.652041 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:49.652056 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:49.652170 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:49.652389 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:49.652392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:49.652400 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:49.652540 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.652547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.653852 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.653856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.654594 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:49.654604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.655216 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.655218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:49.655288 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.655332 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.655636 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.655640 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:43:49.655715 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:43:49.655799 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
I20260928 23:43:49.655959 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:49.651751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:50409 (local address 127.21.112.66:45333)
0928 23:43:49.651876 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:49.651880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:49.652130 (+ 250us) server_negotiation.cc:407] Connection header received
0928 23:43:49.652254 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:49.652256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:49.652296 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:49.652356 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:49.652674 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.653605 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.654717 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:49.655106 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:49.655222 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:49.655463 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:49.655550 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:49.655629 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:49.655725 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:43:49.655795 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:43:49.679160 23686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:49.679415 23686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:49.682394 23686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Sending DeleteTablet for 3 replicas of tablet 423551a9615848e6bf1c57ca3950016c
I20260928 23:43:49.683192 23965 tablet_service.cc:1514] Processing DeleteTablet for tablet 423551a9615848e6bf1c57ca3950016c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:49 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260928 23:43:49.683585 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet 423551a9615848e6bf1c57ca3950016c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:49 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260928 23:43:49.683969 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 423551a9615848e6bf1c57ca3950016c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:49 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:38058
I20260928 23:43:49.684525 24244 tablet_replica.cc:330] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: stopping tablet replica
I20260928 23:43:49.684875 24244 raft_consensus.cc:2229] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:49.684816 23686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:43:49.685279 23686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:49.685305 24244 raft_consensus.cc:2258] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:49.686244 24242 tablet_replica.cc:330] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: stopping tablet replica
I20260928 23:43:49.686573 24242 raft_consensus.cc:2229] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:43:49.687027 24242 raft_consensus.cc:2258] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:49.688532 24242 ts_tablet_manager.cc:1905] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:49.691778 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0103de679f472680ea4df83f4fb18b (DEFAULT_TABLE table=test-table [id=f2d5ebb4470247acadb9699e49a3157c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:49.692067 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0103de679f472680ea4df83f4fb18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.692553 24242 ts_tablet_manager.cc:1918] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:49.692649 24242 log.cc:1196] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/423551a9615848e6bf1c57ca3950016c
I20260928 23:43:49.692987 24242 ts_tablet_manager.cc:1939] T 423551a9615848e6bf1c57ca3950016c P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting consensus metadata
I20260928 23:43:49.693490 24243 tablet_replica.cc:330] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: stopping tablet replica
I20260928 23:43:49.693837 24243 raft_consensus.cc:2229] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:49.694002 24243 raft_consensus.cc:2258] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:49.694694 23965 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0103de679f472680ea4df83f4fb18b (DEFAULT_TABLE table=test-table [id=f2d5ebb4470247acadb9699e49a3157c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:49.694949 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0103de679f472680ea4df83f4fb18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.695178 23674 catalog_manager.cc:4841] TS e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333): tablet 423551a9615848e6bf1c57ca3950016c (table test-table [id=6bde009fca584e15962ca6d031e3cf16]) successfully deleted
I20260928 23:43:49.695386 24243 ts_tablet_manager.cc:1905] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:49.695695 24195 tablet_bootstrap.cc:492] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Bootstrap starting.
I20260928 23:43:49.696677 24195 tablet_bootstrap.cc:654] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.697117 24194 tablet_bootstrap.cc:492] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Bootstrap starting.
I20260928 23:43:49.697501 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 8e0103de679f472680ea4df83f4fb18b (DEFAULT_TABLE table=test-table [id=f2d5ebb4470247acadb9699e49a3157c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:49.697774 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e0103de679f472680ea4df83f4fb18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:49.698056 24194 tablet_bootstrap.cc:654] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.699273 24195 tablet_bootstrap.cc:492] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: No bootstrap required, opened a new log
I20260928 23:43:49.699347 24195 ts_tablet_manager.cc:1397] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 23:43:49.699769 24195 raft_consensus.cc:348] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.699880 24195 raft_consensus.cc:374] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.699919 24195 raft_consensus.cc:729] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8052d79de64b1a9e8179df444d7a4d, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.700078 24196 tablet_bootstrap.cc:492] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Bootstrap starting.
I20260928 23:43:49.700047 24195 consensus_queue.cc:260] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.700320 24195 ts_tablet_manager.cc:1428] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:49.700992 24196 tablet_bootstrap.cc:654] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:49.702778 24194 tablet_bootstrap.cc:492] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: No bootstrap required, opened a new log
I20260928 23:43:49.702865 24194 ts_tablet_manager.cc:1397] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 23:43:49.703302 24194 raft_consensus.cc:348] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.703429 24194 raft_consensus.cc:374] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.703461 24194 raft_consensus.cc:729] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.703573 24194 consensus_queue.cc:260] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.703845 24194 ts_tablet_manager.cc:1428] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:43:49.704842 24244 ts_tablet_manager.cc:1905] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:49.707144 24243 ts_tablet_manager.cc:1918] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:49.707288 24243 log.cc:1196] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/423551a9615848e6bf1c57ca3950016c
I20260928 23:43:49.707640 24243 ts_tablet_manager.cc:1939] T 423551a9615848e6bf1c57ca3950016c P ab0ba9074da6430d89112f9cff7e54b4: Deleting consensus metadata
I20260928 23:43:49.708469 24244 ts_tablet_manager.cc:1918] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:49.708577 24244 log.cc:1196] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/423551a9615848e6bf1c57ca3950016c
I20260928 23:43:49.708830 23673 catalog_manager.cc:4841] TS ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521): tablet 423551a9615848e6bf1c57ca3950016c (table test-table [id=6bde009fca584e15962ca6d031e3cf16]) successfully deleted
I20260928 23:43:49.708942 24244 ts_tablet_manager.cc:1939] T 423551a9615848e6bf1c57ca3950016c P ef8052d79de64b1a9e8179df444d7a4d: Deleting consensus metadata
I20260928 23:43:49.710213 23674 catalog_manager.cc:4841] TS ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875): tablet 423551a9615848e6bf1c57ca3950016c (table test-table [id=6bde009fca584e15962ca6d031e3cf16]) successfully deleted
I20260928 23:43:49.710824 24196 tablet_bootstrap.cc:492] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: No bootstrap required, opened a new log
I20260928 23:43:49.710919 24196 ts_tablet_manager.cc:1397] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260928 23:43:49.711360 24196 raft_consensus.cc:348] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.711493 24196 raft_consensus.cc:374] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:49.711527 24196 raft_consensus.cc:729] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0ba9074da6430d89112f9cff7e54b4, State: Initialized, Role: FOLLOWER
I20260928 23:43:49.711652 24196 consensus_queue.cc:260] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:49.711948 24196 ts_tablet_manager.cc:1428] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:50.060750 24201 raft_consensus.cc:482] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:50.060917 24201 raft_consensus.cc:504] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:50.061316 24201 leader_election.cc:288] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:50.061843 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0103de679f472680ea4df83f4fb18b" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d" is_pre_election: true
I20260928 23:43:50.061859 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0103de679f472680ea4df83f4fb18b" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4" is_pre_election: true
I20260928 23:43:50.062011 23845 raft_consensus.cc:2454] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 0.
I20260928 23:43:50.062029 24125 raft_consensus.cc:2454] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 0.
I20260928 23:43:50.062327 23918 leader_election.cc:302] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, e30ae3be03d04f17a3dd6ba8b18beb2c; no voters:
I20260928 23:43:50.062543 24201 raft_consensus.cc:2781] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:50.062600 24201 raft_consensus.cc:482] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:50.062628 24201 raft_consensus.cc:3037] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:50.063624 24201 raft_consensus.cc:504] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:50.064016 24201 leader_election.cc:288] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 election: Requested vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:50.064386 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0103de679f472680ea4df83f4fb18b" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d"
I20260928 23:43:50.064527 23845 raft_consensus.cc:3037] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:50.064486 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e0103de679f472680ea4df83f4fb18b" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4"
I20260928 23:43:50.064625 24125 raft_consensus.cc:3037] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:50.065678 23845 raft_consensus.cc:2454] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 1.
I20260928 23:43:50.065761 24125 raft_consensus.cc:2454] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 1.
I20260928 23:43:50.065974 23919 leader_election.cc:302] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e30ae3be03d04f17a3dd6ba8b18beb2c, ef8052d79de64b1a9e8179df444d7a4d; no voters:
I20260928 23:43:50.066128 24201 raft_consensus.cc:2781] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:50.066236 24201 raft_consensus.cc:686] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 LEADER]: Becoming Leader. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Running, Role: LEADER
I20260928 23:43:50.066380 24201 consensus_queue.cc:237] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:50.068137 23686 catalog_manager.cc:5495] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c reported cstate change: term changed from 0 to 1, leader changed from <none> to e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66). New cstate: current_term: 1 leader_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } health_report { overall_health: UNKNOWN } } }
I20260928 23:43:50.171444 24125 raft_consensus.cc:1261] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Refusing update from remote peer e30ae3be03d04f17a3dd6ba8b18beb2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:50.171753 23845 raft_consensus.cc:1261] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Refusing update from remote peer e30ae3be03d04f17a3dd6ba8b18beb2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:50.171947 24201 consensus_queue.cc:1035] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:50.172219 24236 consensus_queue.cc:1035] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:43:50.185464 24222 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:43:51.741096 23687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:51.741256 23687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:51.744454 23687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Sending DeleteTablet for 3 replicas of tablet 8e0103de679f472680ea4df83f4fb18b
I20260928 23:43:51.745378 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0103de679f472680ea4df83f4fb18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:51 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:38058
I20260928 23:43:51.745461 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0103de679f472680ea4df83f4fb18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:51 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260928 23:43:51.745455 23965 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e0103de679f472680ea4df83f4fb18b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:51 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260928 23:43:51.745887 24280 tablet_replica.cc:330] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: stopping tablet replica
I20260928 23:43:51.745991 24279 tablet_replica.cc:330] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: stopping tablet replica
I20260928 23:43:51.746050 24280 raft_consensus.cc:2229] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:43:51.746145 24279 raft_consensus.cc:2229] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:51.746264 24279 raft_consensus.cc:2258] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:51.746266 24280 raft_consensus.cc:2258] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:51.746534 24278 tablet_replica.cc:330] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: stopping tablet replica
I20260928 23:43:51.746671 24278 raft_consensus.cc:2229] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:51.746811 24278 raft_consensus.cc:2258] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:51.747282 24280 ts_tablet_manager.cc:1905] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:51.747304 24279 ts_tablet_manager.cc:1905] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:51.747608 24278 ts_tablet_manager.cc:1905] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:51.750026 24280 ts_tablet_manager.cc:1918] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:43:51.750026 24279 ts_tablet_manager.cc:1918] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:43:51.750114 24280 log.cc:1196] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/8e0103de679f472680ea4df83f4fb18b
I20260928 23:43:51.750114 24279 log.cc:1196] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/8e0103de679f472680ea4df83f4fb18b
I20260928 23:43:51.750396 24279 ts_tablet_manager.cc:1939] T 8e0103de679f472680ea4df83f4fb18b P ab0ba9074da6430d89112f9cff7e54b4: Deleting consensus metadata
I20260928 23:43:51.750406 24280 ts_tablet_manager.cc:1939] T 8e0103de679f472680ea4df83f4fb18b P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting consensus metadata
I20260928 23:43:51.751080 24278 ts_tablet_manager.cc:1918] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:43:51.751204 24278 log.cc:1196] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/8e0103de679f472680ea4df83f4fb18b
I20260928 23:43:51.751154 23673 catalog_manager.cc:4841] TS ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521): tablet 8e0103de679f472680ea4df83f4fb18b (table test-table [id=f2d5ebb4470247acadb9699e49a3157c]) successfully deleted
I20260928 23:43:51.751154 23674 catalog_manager.cc:4841] TS e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333): tablet 8e0103de679f472680ea4df83f4fb18b (table test-table [id=f2d5ebb4470247acadb9699e49a3157c]) successfully deleted
I20260928 23:43:51.751502 24278 ts_tablet_manager.cc:1939] T 8e0103de679f472680ea4df83f4fb18b P ef8052d79de64b1a9e8179df444d7a4d: Deleting consensus metadata
I20260928 23:43:51.752166 23674 catalog_manager.cc:4841] TS ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875): tablet 8e0103de679f472680ea4df83f4fb18b (table test-table [id=f2d5ebb4470247acadb9699e49a3157c]) successfully deleted
Sep 28 23:43:51 dist-test-slave-c3s8 krb5kdc[23642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639031, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:43:51 dist-test-slave-c3s8 krb5kdc[23642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639031, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:51.785470 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.768894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.21.112.126:46221)
0928 23:43:51.769210 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:51.769218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:51.769234 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:43:51.769283 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:51.769285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:51.769337 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:51.769467 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:51.770475 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.771081 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.772053 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.772336 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.772434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:51.781958 (+ 9524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:51.781963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:51.781965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:51.782000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:51.783744 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:51.784192 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:51.784194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:51.784196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:51.784272 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:51.784593 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:51.784595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:51.784596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:51.784782 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:51.784881 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:51.785123 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:43:51.785229 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":144,"threads_started":1}
I20260928 23:43:51.788249 23687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:43:51.788740 23687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:51.794080 23825 tablet_service.cc:1467] Processing CreateTablet for tablet 69993a16c156467292c59e021e58bbab (DEFAULT_TABLE table=test-table [id=6aa1654fffaf4cb38fbfc6f52610fc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:51.794109 23965 tablet_service.cc:1467] Processing CreateTablet for tablet 69993a16c156467292c59e021e58bbab (DEFAULT_TABLE table=test-table [id=6aa1654fffaf4cb38fbfc6f52610fc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:51.794401 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69993a16c156467292c59e021e58bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:51.794421 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69993a16c156467292c59e021e58bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:51.797004 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 69993a16c156467292c59e021e58bbab (DEFAULT_TABLE table=test-table [id=6aa1654fffaf4cb38fbfc6f52610fc4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:51.797353 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69993a16c156467292c59e021e58bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:51.797736 24289 tablet_bootstrap.cc:492] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Bootstrap starting.
I20260928 23:43:51.798544 24289 tablet_bootstrap.cc:654] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:51.799146 24290 tablet_bootstrap.cc:492] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Bootstrap starting.
I20260928 23:43:51.799746 24289 tablet_bootstrap.cc:492] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: No bootstrap required, opened a new log
I20260928 23:43:51.799837 24289 ts_tablet_manager.cc:1397] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:51.799950 24292 tablet_bootstrap.cc:492] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Bootstrap starting.
I20260928 23:43:51.800133 24290 tablet_bootstrap.cc:654] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:51.800290 24289 raft_consensus.cc:348] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.800421 24289 raft_consensus.cc:374] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:51.800460 24289 raft_consensus.cc:729] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Initialized, Role: FOLLOWER
I20260928 23:43:51.800599 24289 consensus_queue.cc:260] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.800894 24289 ts_tablet_manager.cc:1428] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:43:51.800951 24292 tablet_bootstrap.cc:654] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:51.801975 24290 tablet_bootstrap.cc:492] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: No bootstrap required, opened a new log
I20260928 23:43:51.802089 24290 ts_tablet_manager.cc:1397] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:43:51.802212 24292 tablet_bootstrap.cc:492] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: No bootstrap required, opened a new log
I20260928 23:43:51.802305 24292 ts_tablet_manager.cc:1397] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:43:51.802642 24290 raft_consensus.cc:348] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.802763 24290 raft_consensus.cc:374] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:51.802798 24290 raft_consensus.cc:729] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8052d79de64b1a9e8179df444d7a4d, State: Initialized, Role: FOLLOWER
I20260928 23:43:51.802754 24292 raft_consensus.cc:348] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.802867 24292 raft_consensus.cc:374] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:51.802927 24292 raft_consensus.cc:729] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0ba9074da6430d89112f9cff7e54b4, State: Initialized, Role: FOLLOWER
I20260928 23:43:51.802923 24290 consensus_queue.cc:260] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.803071 24292 consensus_queue.cc:260] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.803390 24292 ts_tablet_manager.cc:1428] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:43:51.803836 24290 ts_tablet_manager.cc:1428] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:51.868748 24256 raft_consensus.cc:482] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:51.868927 24256 raft_consensus.cc:504] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.869323 24256 leader_election.cc:288] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:51.869999 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69993a16c156467292c59e021e58bbab" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d" is_pre_election: true
I20260928 23:43:51.869976 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69993a16c156467292c59e021e58bbab" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4" is_pre_election: true
I20260928 23:43:51.870151 24125 raft_consensus.cc:2454] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 0.
I20260928 23:43:51.870151 23845 raft_consensus.cc:2454] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 0.
I20260928 23:43:51.870589 23918 leader_election.cc:302] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, e30ae3be03d04f17a3dd6ba8b18beb2c; no voters:
I20260928 23:43:51.870819 24256 raft_consensus.cc:2781] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:51.870898 24256 raft_consensus.cc:482] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:51.870951 24256 raft_consensus.cc:3037] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:51.872154 24256 raft_consensus.cc:504] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.872566 24256 leader_election.cc:288] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 election: Requested vote from peers ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:51.873024 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69993a16c156467292c59e021e58bbab" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4"
I20260928 23:43:51.873054 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69993a16c156467292c59e021e58bbab" candidate_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8052d79de64b1a9e8179df444d7a4d"
I20260928 23:43:51.873174 23845 raft_consensus.cc:3037] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:51.873174 24125 raft_consensus.cc:3037] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:51.874425 23845 raft_consensus.cc:2454] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 1.
I20260928 23:43:51.874459 24125 raft_consensus.cc:2454] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30ae3be03d04f17a3dd6ba8b18beb2c in term 1.
I20260928 23:43:51.874766 23918 leader_election.cc:302] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, e30ae3be03d04f17a3dd6ba8b18beb2c; no voters:
I20260928 23:43:51.875023 24256 raft_consensus.cc:2781] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:51.875142 24256 raft_consensus.cc:686] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 LEADER]: Becoming Leader. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Running, Role: LEADER
I20260928 23:43:51.875315 24256 consensus_queue.cc:237] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.877153 23685 catalog_manager.cc:5495] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c reported cstate change: term changed from 0 to 1, leader changed from <none> to e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66). New cstate: current_term: 1 leader_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } health_report { overall_health: UNKNOWN } } }
I20260928 23:43:51.908311 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.904956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.21.112.66:45333)
0928 23:43:51.905238 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:51.905245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:51.905256 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:43:51.905292 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:51.905294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:51.905334 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:51.905435 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:51.905779 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.906368 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.907236 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.907472 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.907586 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:51.907849 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:51.907928 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:51.908011 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:51.908100 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:43:51.908152 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260928 23:43:51.914966 24125 raft_consensus.cc:1261] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Refusing update from remote peer e30ae3be03d04f17a3dd6ba8b18beb2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:51.915323 23845 raft_consensus.cc:1261] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Refusing update from remote peer e30ae3be03d04f17a3dd6ba8b18beb2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:51.915354 24269 consensus_queue.cc:1035] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:51.915736 24269 consensus_queue.cc:1035] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:51.949988 23686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:51.950152 23686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:51.952247 23686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Sending DeleteTablet for 3 replicas of tablet 69993a16c156467292c59e021e58bbab
I20260928 23:43:51.952950 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet 69993a16c156467292c59e021e58bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:51 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260928 23:43:51.952950 23965 tablet_service.cc:1514] Processing DeleteTablet for tablet 69993a16c156467292c59e021e58bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:51 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260928 23:43:51.953001 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet 69993a16c156467292c59e021e58bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:51 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:38058
I20260928 23:43:51.953157 24279 tablet_replica.cc:330] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: stopping tablet replica
I20260928 23:43:51.953178 24280 tablet_replica.cc:330] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: stopping tablet replica
I20260928 23:43:51.953233 24278 tablet_replica.cc:330] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: stopping tablet replica
I20260928 23:43:51.953284 24280 raft_consensus.cc:2229] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:43:51.953282 24279 raft_consensus.cc:2229] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:51.953369 24278 raft_consensus.cc:2229] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:51.953405 24279 raft_consensus.cc:2258] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:51.953498 24280 raft_consensus.cc:2258] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:51.953522 24278 raft_consensus.cc:2258] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:51.954212 24279 ts_tablet_manager.cc:1905] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:51.954268 24280 ts_tablet_manager.cc:1905] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:51.954411 24278 ts_tablet_manager.cc:1905] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:51.954519 23686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:43:51.954973 23686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:43:51.956743 24279 ts_tablet_manager.cc:1918] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:51.956744 24280 ts_tablet_manager.cc:1918] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:51.956815 24279 log.cc:1196] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/69993a16c156467292c59e021e58bbab
I20260928 23:43:51.956815 24280 log.cc:1196] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/69993a16c156467292c59e021e58bbab
I20260928 23:43:51.956933 24278 ts_tablet_manager.cc:1918] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:43:51.956990 24278 log.cc:1196] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/69993a16c156467292c59e021e58bbab
I20260928 23:43:51.957077 24280 ts_tablet_manager.cc:1939] T 69993a16c156467292c59e021e58bbab P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting consensus metadata
I20260928 23:43:51.957077 24279 ts_tablet_manager.cc:1939] T 69993a16c156467292c59e021e58bbab P ab0ba9074da6430d89112f9cff7e54b4: Deleting consensus metadata
I20260928 23:43:51.957193 24278 ts_tablet_manager.cc:1939] T 69993a16c156467292c59e021e58bbab P ef8052d79de64b1a9e8179df444d7a4d: Deleting consensus metadata
I20260928 23:43:51.958048 23674 catalog_manager.cc:4841] TS ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875): tablet 69993a16c156467292c59e021e58bbab (table test-table [id=6aa1654fffaf4cb38fbfc6f52610fc4d]) successfully deleted
I20260928 23:43:51.958232 23674 catalog_manager.cc:4841] TS e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333): tablet 69993a16c156467292c59e021e58bbab (table test-table [id=6aa1654fffaf4cb38fbfc6f52610fc4d]) successfully deleted
I20260928 23:43:51.958266 23673 catalog_manager.cc:4841] TS ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521): tablet 69993a16c156467292c59e021e58bbab (table test-table [id=6aa1654fffaf4cb38fbfc6f52610fc4d]) successfully deleted
I20260928 23:43:51.961297 23825 tablet_service.cc:1467] Processing CreateTablet for tablet baed4e1038e045a591ab03d0021f8b4a (DEFAULT_TABLE table=test-table [id=c1ad859aea3f4f79a58b4c7d313becce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:51.961651 23825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baed4e1038e045a591ab03d0021f8b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:51.961628 24105 tablet_service.cc:1467] Processing CreateTablet for tablet baed4e1038e045a591ab03d0021f8b4a (DEFAULT_TABLE table=test-table [id=c1ad859aea3f4f79a58b4c7d313becce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:51.961875 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baed4e1038e045a591ab03d0021f8b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:51.961827 23965 tablet_service.cc:1467] Processing CreateTablet for tablet baed4e1038e045a591ab03d0021f8b4a (DEFAULT_TABLE table=test-table [id=c1ad859aea3f4f79a58b4c7d313becce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:43:51.962105 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baed4e1038e045a591ab03d0021f8b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:51.964032 24290 tablet_bootstrap.cc:492] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d: Bootstrap starting.
I20260928 23:43:51.964241 24289 tablet_bootstrap.cc:492] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Bootstrap starting.
I20260928 23:43:51.965279 24290 tablet_bootstrap.cc:654] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:51.965651 24292 tablet_bootstrap.cc:492] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Bootstrap starting.
I20260928 23:43:51.965902 24289 tablet_bootstrap.cc:654] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:51.966706 24292 tablet_bootstrap.cc:654] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:51.966722 24290 tablet_bootstrap.cc:492] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d: No bootstrap required, opened a new log
I20260928 23:43:51.966809 24290 ts_tablet_manager.cc:1397] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:43:51.967069 24289 tablet_bootstrap.cc:492] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: No bootstrap required, opened a new log
I20260928 23:43:51.967175 24289 ts_tablet_manager.cc:1397] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:43:51.967332 24290 raft_consensus.cc:348] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.967483 24290 raft_consensus.cc:374] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:51.967531 24290 raft_consensus.cc:729] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8052d79de64b1a9e8179df444d7a4d, State: Initialized, Role: FOLLOWER
I20260928 23:43:51.967581 24289 raft_consensus.cc:348] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.967737 24289 raft_consensus.cc:374] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:51.967713 24290 consensus_queue.cc:260] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.967774 24289 raft_consensus.cc:729] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30ae3be03d04f17a3dd6ba8b18beb2c, State: Initialized, Role: FOLLOWER
I20260928 23:43:51.967938 24292 tablet_bootstrap.cc:492] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: No bootstrap required, opened a new log
I20260928 23:43:51.967921 24289 consensus_queue.cc:260] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.968019 24292 ts_tablet_manager.cc:1397] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:51.968444 24292 raft_consensus.cc:348] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.968530 24290 ts_tablet_manager.cc:1428] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:43:51.968568 24292 raft_consensus.cc:374] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:51.968602 24292 raft_consensus.cc:729] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0ba9074da6430d89112f9cff7e54b4, State: Initialized, Role: FOLLOWER
I20260928 23:43:51.968735 24292 consensus_queue.cc:260] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.969041 24292 ts_tablet_manager.cc:1428] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:43:51.969808 24289 ts_tablet_manager.cc:1428] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:43:51.969956 24295 raft_consensus.cc:482] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:43:51.970064 24295 raft_consensus.cc:504] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.971356 24295 leader_election.cc:288] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:51.976311 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.971898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:41777 (local address 127.21.112.67:36521)
0928 23:43:51.972266 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:51.972280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:51.972301 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:43:51.972366 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:51.972371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:51.972418 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:51.972506 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:51.972898 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.973660 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.975078 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.975685 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.975807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:51.975871 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:51.976090 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 23:43:51.976173 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":70,"threads_started":1}
I20260928 23:43:51.976367 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.971569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:36521 (local address 127.21.112.65:41777)
0928 23:43:51.972006 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:51.972024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:51.972138 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:51.972626 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:51.972629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:51.972644 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:51.972764 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:51.972770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.973807 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.973811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:51.974976 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:51.974986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.975805 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.975808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:51.975938 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:51.975961 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:43:51.976075 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:43:51.976155 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":74,"threads_started":1}
I20260928 23:43:51.976823 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baed4e1038e045a591ab03d0021f8b4a" candidate_uuid: "ef8052d79de64b1a9e8179df444d7a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4" is_pre_election: true
I20260928 23:43:51.976995 24125 raft_consensus.cc:2454] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef8052d79de64b1a9e8179df444d7a4d in term 0.
I20260928 23:43:51.977346 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.971683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45333 (local address 127.21.112.65:33893)
0928 23:43:51.973314 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:51.973326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:51.973441 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:51.973727 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:51.973730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:51.973746 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:43:51.973867 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:51.973876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.974723 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.974725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:51.975841 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:51.975850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.976939 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.976942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:51.977042 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:51.977063 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:43:51.977145 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:43:51.977215 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1576,"thread_start_us":100,"threads_started":1}
I20260928 23:43:51.977375 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.971998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:33893 (local address 127.21.112.66:45333)
0928 23:43:51.972139 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:51.972145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:51.973409 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 23:43:51.973584 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:51.973587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:51.973632 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:51.973718 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:43:51.974000 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.974594 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.976005 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.976727 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:51.976946 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:51.976995 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:51.977168 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:43:51.977259 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:43:51.977463 23778 leader_election.cc:302] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0ba9074da6430d89112f9cff7e54b4, ef8052d79de64b1a9e8179df444d7a4d; no voters:
I20260928 23:43:51.977890 23985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baed4e1038e045a591ab03d0021f8b4a" candidate_uuid: "ef8052d79de64b1a9e8179df444d7a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" is_pre_election: true
I20260928 23:43:51.977811 24295 raft_consensus.cc:2781] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:43:51.978040 24295 raft_consensus.cc:482] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:43:51.978065 23985 raft_consensus.cc:2454] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef8052d79de64b1a9e8179df444d7a4d in term 0.
I20260928 23:43:51.978127 24295 raft_consensus.cc:3037] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:51.979246 24295 raft_consensus.cc:504] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.979658 24295 leader_election.cc:288] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [CANDIDATE]: Term 1 election: Requested vote from peers e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333), ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521)
I20260928 23:43:51.979969 23985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baed4e1038e045a591ab03d0021f8b4a" candidate_uuid: "ef8052d79de64b1a9e8179df444d7a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c"
I20260928 23:43:51.979995 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baed4e1038e045a591ab03d0021f8b4a" candidate_uuid: "ef8052d79de64b1a9e8179df444d7a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0ba9074da6430d89112f9cff7e54b4"
I20260928 23:43:51.980077 23985 raft_consensus.cc:3037] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:51.980121 24125 raft_consensus.cc:3037] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:51.980979 23985 raft_consensus.cc:2454] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef8052d79de64b1a9e8179df444d7a4d in term 1.
I20260928 23:43:51.981101 24125 raft_consensus.cc:2454] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef8052d79de64b1a9e8179df444d7a4d in term 1.
I20260928 23:43:51.981390 23779 leader_election.cc:302] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e30ae3be03d04f17a3dd6ba8b18beb2c, ef8052d79de64b1a9e8179df444d7a4d; no voters:
I20260928 23:43:51.981629 24295 raft_consensus.cc:2781] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:51.981959 24295 raft_consensus.cc:686] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [term 1 LEADER]: Becoming Leader. State: Replica: ef8052d79de64b1a9e8179df444d7a4d, State: Running, Role: LEADER
I20260928 23:43:51.982326 24295 consensus_queue.cc:237] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } }
I20260928 23:43:51.984755 23686 catalog_manager.cc:5495] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65). New cstate: current_term: 1 leader_uuid: "ef8052d79de64b1a9e8179df444d7a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8052d79de64b1a9e8179df444d7a4d" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 32875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 } health_report { overall_health: UNKNOWN } } }
I20260928 23:43:52.002233 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:51.998043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.21.112.65:32875)
0928 23:43:51.998392 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:51.998399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:51.998410 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:43:51.998478 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:51.998480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:51.998530 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:51.998658 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:51.998959 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:51.999521 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.000355 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.000654 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.000765 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:52.001053 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:52.001599 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:52.001734 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:52.001800 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 23:43:52.001855 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
I20260928 23:43:52.014843 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:52.009315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:36521 (local address 127.21.112.65:39095)
0928 23:43:52.009678 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:52.009691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:52.009844 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:52.010217 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:52.010223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:52.010237 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:52.010395 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:52.010404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.011765 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.011772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:52.013049 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:52.013064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.013852 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.013858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:52.014000 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:52.014063 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:52.014441 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:52.014452 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 23:43:52.014542 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:43:52.014636 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260928 23:43:52.015605 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:52.009417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:39095 (local address 127.21.112.67:36521)
0928 23:43:52.009612 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:52.009617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:52.009759 (+ 142us) server_negotiation.cc:407] Connection header received
0928 23:43:52.010007 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:52.010014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:52.010079 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:52.010150 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:52.010533 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.011629 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.013198 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.013694 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.013852 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:52.014257 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:52.014350 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:52.014444 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:52.014542 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:43:52.015418 (+ 876us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:43:52.023254 23985 raft_consensus.cc:1261] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Refusing update from remote peer ef8052d79de64b1a9e8179df444d7a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:52.023283 24125 raft_consensus.cc:1261] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Refusing update from remote peer ef8052d79de64b1a9e8179df444d7a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:43:52.023809 24295 consensus_queue.cc:1035] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0ba9074da6430d89112f9cff7e54b4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:52.024072 24309 consensus_queue.cc:1035] T baed4e1038e045a591ab03d0021f8b4a P ef8052d79de64b1a9e8179df444d7a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30ae3be03d04f17a3dd6ba8b18beb2c" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:43:52.049585 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:52.046198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:50769 (local address 127.21.112.65:32875)
0928 23:43:52.046485 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:52.046493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:52.046508 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:43:52.046675 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:52.046678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:52.046733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:52.046804 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:43:52.047074 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.047639 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.048498 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.048740 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.048862 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:52.049109 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:43:52.049192 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:43:52.049271 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:52.049360 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:43:52.049419 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260928 23:43:52.049605 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:52.046115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:32875 (local address 127.21.112.67:50769)
0928 23:43:52.046423 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:52.046441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:52.046542 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:52.046829 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:52.046835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:52.046846 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:43:52.046965 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:52.046975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.047751 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.047757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:52.048374 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:52.048382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:52.048853 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:52.048856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:52.048946 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:52.048981 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:43:52.049273 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:43:52.049282 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:43:52.049358 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:43:52.049453 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":139,"threads_started":1}
W20260928 23:43:52.050375 24260 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:43:53.626122 23685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:53.626298 23685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:43:53.629415 23685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc3f2ac0f4b441a80ab83482d1e0bde: Sending DeleteTablet for 3 replicas of tablet baed4e1038e045a591ab03d0021f8b4a
I20260928 23:43:53.630095 23825 tablet_service.cc:1514] Processing DeleteTablet for tablet baed4e1038e045a591ab03d0021f8b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:53 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:38058
I20260928 23:43:53.630167 23965 tablet_service.cc:1514] Processing DeleteTablet for tablet baed4e1038e045a591ab03d0021f8b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:53 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37764
I20260928 23:43:53.630380 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet baed4e1038e045a591ab03d0021f8b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:43:53 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260928 23:43:53.630520 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23760
I20260928 23:43:53.630558 24343 tablet_replica.cc:330] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: stopping tablet replica
I20260928 23:43:53.631511 24343 raft_consensus.cc:2229] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:53.631651 24344 tablet_replica.cc:330] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: stopping tablet replica
I20260928 23:43:53.631666 24343 raft_consensus.cc:2258] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:53.631868 24344 raft_consensus.cc:2229] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:43:53.632046 24344 raft_consensus.cc:2258] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:43:53.632380 24343 ts_tablet_manager.cc:1905] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:53.633221 24344 ts_tablet_manager.cc:1905] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:43:53.635437 24344 ts_tablet_manager.cc:1918] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:43:53.635541 24344 log.cc:1196] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/baed4e1038e045a591ab03d0021f8b4a
I20260928 23:43:53.635787 24344 ts_tablet_manager.cc:1939] T baed4e1038e045a591ab03d0021f8b4a P ab0ba9074da6430d89112f9cff7e54b4: Deleting consensus metadata
I20260928 23:43:53.635885 24343 ts_tablet_manager.cc:1918] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:43:53.635972 24343 log.cc:1196] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/baed4e1038e045a591ab03d0021f8b4a
I20260928 23:43:53.636255 24343 ts_tablet_manager.cc:1939] T baed4e1038e045a591ab03d0021f8b4a P e30ae3be03d04f17a3dd6ba8b18beb2c: Deleting consensus metadata
I20260928 23:43:53.636546 23673 catalog_manager.cc:4841] TS ab0ba9074da6430d89112f9cff7e54b4 (127.21.112.67:36521): tablet baed4e1038e045a591ab03d0021f8b4a (table test-table [id=c1ad859aea3f4f79a58b4c7d313becce]) successfully deleted
I20260928 23:43:53.637094 23674 catalog_manager.cc:4841] TS e30ae3be03d04f17a3dd6ba8b18beb2c (127.21.112.66:45333): tablet baed4e1038e045a591ab03d0021f8b4a (table test-table [id=c1ad859aea3f4f79a58b4c7d313becce]) successfully deleted
W20260928 23:43:53.639643 23674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:32875 (error 108)
I20260928 23:43:53.639896 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23900
I20260928 23:43:53.641675 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:53.641211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:32875 (local address 127.0.0.1:43946)
0928 23:43:53.641463 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:53.641573 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
W20260928 23:43:53.641896 23674 catalog_manager.cc:4584] TS ef8052d79de64b1a9e8179df444d7a4d (127.21.112.65:32875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet baed4e1038e045a591ab03d0021f8b4a: Network error: Client connection negotiation failed: client connection to 127.21.112.65:32875: connect: Connection refused (error 111)
I20260928 23:43:53.647465 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24040
I20260928 23:43:53.654129 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 23655
2026-09-28T23:43:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6787 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:53 dist-test-slave-c3s8 krb5kdc[24349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:53 dist-test-slave-c3s8 krb5kdc[24349](info): set up 2 sockets
Sep 28 23:43:53 dist-test-slave-c3s8 krb5kdc[24349](info): commencing operation
krb5kdc: starting...
W20260928 23:43:54.742058 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:43:54 dist-test-slave-c3s8 krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:43:54.869242 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:39347
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:54.983685 24362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:54.984050 24362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:54.984140 24362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:54.988371 24362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:54.988468 24362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:54.988505 24362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:54.988544 24362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:54.988581 24362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:54.994159 24362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:39347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:54.995311 24362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:54.996366 24362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:55.005760 24368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:55.005904 24362 server_base.cc:1034] running on GCE node
W20260928 23:43:55.005779 24367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:55.006129 24370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:55.006722 24362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:55.008052 24362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:55.009259 24362 hybrid_clock.cc:648] HybridClock initialized: now 1790639035009239 us; error 50 us; skew 500 ppm
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:55.012324 24362 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:55.013372 24362 webserver.cc:460] Webserver started at http://127.21.112.126:40315/ using document root <none> and password file <none>
I20260928 23:43:55.014035 24362 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:55.014132 24362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:55.014375 24362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:55.016573 24362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "6f542961a6b04a52bddb27cb64c389bc"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "12d796bb1e1d358768ff8094c60f867d"
server_key_iv: "df3eb3591ba5e963c5e3a128f1ea2f0a"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.017158 24362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "6f542961a6b04a52bddb27cb64c389bc"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "12d796bb1e1d358768ff8094c60f867d"
server_key_iv: "df3eb3591ba5e963c5e3a128f1ea2f0a"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.021060 24362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:43:55.024034 24377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.025473 24362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:43:55.025678 24362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "6f542961a6b04a52bddb27cb64c389bc"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "12d796bb1e1d358768ff8094c60f867d"
server_key_iv: "df3eb3591ba5e963c5e3a128f1ea2f0a"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.025828 24362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:55.047613 24362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:55.051256 24362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:55.051496 24362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:55.059229 24362 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:39347
I20260928 23:43:55.059250 24429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:39347 every 8 connection(s)
I20260928 23:43:55.060285 24362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:55.063285 24430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:55.066854 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24362
I20260928 23:43:55.066989 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:55.067261 21953 external_mini_cluster.cc:1428] Setting key 38fdbc9134371fad42d5aabeec25ac57
I20260928 23:43:55.069767 24430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc: Bootstrap starting.
I20260928 23:43:55.072126 24430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:55.072928 24430 log.cc:824] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:55.074987 24430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc: No bootstrap required, opened a new log
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:55.079802 24430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f542961a6b04a52bddb27cb64c389bc" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 39347 } }
I20260928 23:43:55.080119 24430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:55.080174 24430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f542961a6b04a52bddb27cb64c389bc, State: Initialized, Role: FOLLOWER
I20260928 23:43:55.080646 24430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f542961a6b04a52bddb27cb64c389bc" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 39347 } }
I20260928 23:43:55.080816 24430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:55.080914 24430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:55.081007 24430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:55.082325 24430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f542961a6b04a52bddb27cb64c389bc" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 39347 } }
I20260928 23:43:55.082705 24430 leader_election.cc:302] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f542961a6b04a52bddb27cb64c389bc; no voters:
I20260928 23:43:55.083078 24430 leader_election.cc:288] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:55.083252 24435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:55.083518 24435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [term 1 LEADER]: Becoming Leader. State: Replica: 6f542961a6b04a52bddb27cb64c389bc, State: Running, Role: LEADER
I20260928 23:43:55.083783 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.068827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.21.112.126:39347)
0928 23:43:55.069373 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:55.069390 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:55.069431 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:43:55.070302 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:55.070323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:55.070705 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:55.071097 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:55.072042 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.073289 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.074369 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.074941 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.075144 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.078016 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:55.078025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:55.078052 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:55.078097 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:55.080243 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.080788 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.080794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.080800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.080983 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.081283 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.081289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.081293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.081864 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:55.082076 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:55.082376 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 23:43:55.082702 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":146,"threads_started":1}
I20260928 23:43:55.083880 24435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f542961a6b04a52bddb27cb64c389bc" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 39347 } }
I20260928 23:43:55.084422 24430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:43:55.085076 24435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f542961a6b04a52bddb27cb64c389bc. Latest consensus state: current_term: 1 leader_uuid: "6f542961a6b04a52bddb27cb64c389bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f542961a6b04a52bddb27cb64c389bc" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 39347 } } }
I20260928 23:43:55.085209 24435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:55.086827 24442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:55.087118 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f542961a6b04a52bddb27cb64c389bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f542961a6b04a52bddb27cb64c389bc" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 39347 } } }
I20260928 23:43:55.087307 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:55.089846 24442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:55.095420 24442 catalog_manager.cc:1347] Generated new cluster ID: 2d8d9e375a394466bbab7654eb03ebbd
I20260928 23:43:55.095522 24442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:55.108371 24442 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:55.109476 24442 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:55.115137 24442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f542961a6b04a52bddb27cb64c389bc: Generated new TSK 0
I20260928 23:43:55.115896 24442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:43:55.149303 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:39347
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:55.266986 24456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:55.267272 24456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:55.267323 24456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:55.271476 24456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:55.271555 24456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:55.271617 24456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:43:55.277266 24456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:55.278439 24456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:55.279466 24456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:55.289311 24462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:55.289306 24461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:55.289460 24456 server_base.cc:1034] running on GCE node
W20260928 23:43:55.289676 24464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:55.290125 24456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:55.290753 24456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:55.291965 24456 hybrid_clock.cc:648] HybridClock initialized: now 1790639035291922 us; error 51 us; skew 500 ppm
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:55.295035 24456 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:43:55.296012 24456 webserver.cc:460] Webserver started at http://127.21.112.65:44217/ using document root <none> and password file <none>
I20260928 23:43:55.296602 24456 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:55.296694 24456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:55.296926 24456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:55.299081 24456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "b185b37e85254bda83d20d1c3126dd52"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "42d1d20583c85156bbe83d9f011fb000"
server_key_iv: "88b133c849a828d15217406245d571f9"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.299669 24456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "b185b37e85254bda83d20d1c3126dd52"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "42d1d20583c85156bbe83d9f011fb000"
server_key_iv: "88b133c849a828d15217406245d571f9"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.303642 24456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:43:55.306192 24471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.307466 24456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:43:55.307588 24456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "b185b37e85254bda83d20d1c3126dd52"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "42d1d20583c85156bbe83d9f011fb000"
server_key_iv: "88b133c849a828d15217406245d571f9"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.307696 24456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:55.336894 24456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:55.340018 24456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:55.340220 24456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:55.340834 24456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:55.341735 24456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:55.341784 24456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.341854 24456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:55.341884 24456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.350543 24456 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:36875
I20260928 23:43:55.350600 24584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:36875 every 8 connection(s)
I20260928 23:43:55.351500 24456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:43:55.356649 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24456
I20260928 23:43:55.356779 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:43:55.357155 21953 external_mini_cluster.cc:1428] Setting key 68fbf82fa9e27b7c91c217b52b359a2a
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:55.366262 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.353449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:51205 (local address 127.21.112.126:39347)
0928 23:43:55.353635 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:55.353639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:55.354594 (+ 955us) server_negotiation.cc:407] Connection header received
0928 23:43:55.355499 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:55.355502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:55.355561 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:55.355663 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:55.356952 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.357754 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.359264 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.359692 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.359816 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.362409 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:55.362412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:55.362415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:55.362447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:55.363891 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.364568 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.364571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.364572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.364648 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.365057 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.365059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.365060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.365239 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:55.365316 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:55.366019 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 23:43:55.366124 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:43:55.367486 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.353792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:39347 (local address 127.21.112.65:51205)
0928 23:43:55.354448 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:55.354488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:55.355290 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:55.355849 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:55.355855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:55.356203 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:55.356727 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:55.356748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.357977 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.357981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:55.359136 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:55.359146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.359880 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.359884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:55.360027 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.360675 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:55.360691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:55.362232 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:55.364018 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:55.364027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:55.364047 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:55.364464 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:55.364742 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:55.364745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:55.364747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:55.364954 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:55.365354 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:55.365361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:55.365741 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 23:43:55.365966 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 23:43:55.366342 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":110,"threads_started":1}
I20260928 23:43:55.368757 24585 heartbeater.cc:347] Connected to a master server at 127.21.112.126:39347
I20260928 23:43:55.369083 24585 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:43:55.369762 24585 heartbeater.cc:511] Master 127.21.112.126:39347 requested a full tablet report, sending...
I20260928 23:43:55.371361 24394 ts_manager.cc:194] Registered new tserver with Master: b185b37e85254bda83d20d1c3126dd52 (127.21.112.65:36875)
I20260928 23:43:55.372499 24394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:51205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:43:55.386179 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:39347
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:55.500227 24590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:55.500484 24590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:55.500526 24590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:55.504554 24590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:55.504618 24590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:55.504715 24590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:43:55.510133 24590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:55.511279 24590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:55.512375 24590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:55.522014 24595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:55.522071 24598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:55.522081 24596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:55.522153 24590 server_base.cc:1034] running on GCE node
I20260928 23:43:55.522718 24590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:55.523375 24590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:55.524580 24590 hybrid_clock.cc:648] HybridClock initialized: now 1790639035524559 us; error 29 us; skew 500 ppm
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:55.527438 24590 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:43:55.528324 24590 webserver.cc:460] Webserver started at http://127.21.112.66:43487/ using document root <none> and password file <none>
I20260928 23:43:55.529034 24590 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:55.529125 24590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:55.529351 24590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:55.531621 24590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "9d794eeadef44c30968e05c19acbf9c6"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "5f8cc911752149713b2f90f4ca649853"
server_key_iv: "a2b2b20c7e47b3d61cb9f3b79e693bd3"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.532198 24590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "9d794eeadef44c30968e05c19acbf9c6"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "5f8cc911752149713b2f90f4ca649853"
server_key_iv: "a2b2b20c7e47b3d61cb9f3b79e693bd3"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.536144 24590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:43:55.538738 24605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.540092 24590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:43:55.540234 24590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "9d794eeadef44c30968e05c19acbf9c6"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "5f8cc911752149713b2f90f4ca649853"
server_key_iv: "a2b2b20c7e47b3d61cb9f3b79e693bd3"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.540359 24590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:55.563172 24590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:55.566620 24590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:55.566845 24590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:55.567485 24590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:55.568375 24590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:55.568449 24590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.568523 24590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:55.568574 24590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.577723 24590 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:45161
I20260928 23:43:55.577740 24718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:45161 every 8 connection(s)
I20260928 23:43:55.578634 24590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:43:55.582823 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24590
I20260928 23:43:55.582954 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:43:55.583221 21953 external_mini_cluster.cc:1428] Setting key 75a6e33b5f0b635b1105badee04eb279
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:43:55.593130 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.580421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:59967 (local address 127.21.112.126:39347)
0928 23:43:55.580578 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:55.580582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:55.581600 (+ 1018us) server_negotiation.cc:407] Connection header received
0928 23:43:55.582519 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:55.582523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:55.582581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:55.582684 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:55.584175 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.584928 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.586409 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.586640 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.586770 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.589433 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:55.589437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:55.589439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:55.589463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:55.590957 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.591474 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.591478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.591480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.591562 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.591902 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.591907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.591909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.592091 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:55.592199 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:55.592750 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 23:43:55.592961 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:43:55.594096 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.580781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:39347 (local address 127.21.112.66:59967)
0928 23:43:55.581424 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:55.581464 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:55.582300 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:55.582948 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:55.582960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:55.583430 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:55.584033 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:55.584048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.585061 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.585064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:55.586221 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:55.586230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.586875 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.586880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:55.587031 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.587714 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:55.587732 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:55.589252 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:55.591077 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:55.591083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:55.591099 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:55.591370 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:55.591656 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:55.591660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:55.591662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:55.591806 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:55.592197 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:55.592204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:55.592500 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 23:43:55.592707 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 23:43:55.593062 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:43:55.595369 24719 heartbeater.cc:347] Connected to a master server at 127.21.112.126:39347
I20260928 23:43:55.595662 24719 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:55.596160 24719 heartbeater.cc:511] Master 127.21.112.126:39347 requested a full tablet report, sending...
I20260928 23:43:55.597419 24394 ts_manager.cc:194] Registered new tserver with Master: 9d794eeadef44c30968e05c19acbf9c6 (127.21.112.66:45161)
I20260928 23:43:55.598050 24394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:59967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:43:55.613782 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:39347
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:55.726358 24724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:55.726665 24724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:55.726737 24724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:55.731086 24724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:55.731178 24724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:55.731278 24724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:43:55.736976 24724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:39347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:55.738215 24724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:55.739315 24724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:55.749907 24732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:55.749938 24729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:55.749948 24730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:55.750070 24724 server_base.cc:1034] running on GCE node
I20260928 23:43:55.750725 24724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:55.751498 24724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:55.752707 24724 hybrid_clock.cc:648] HybridClock initialized: now 1790639035752684 us; error 38 us; skew 500 ppm
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:55.755821 24724 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:43:55.756798 24724 webserver.cc:460] Webserver started at http://127.21.112.67:37485/ using document root <none> and password file <none>
I20260928 23:43:55.757406 24724 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:55.757510 24724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:55.757788 24724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:55.760062 24724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "8b33505f8d6e417b9724193be517ae5a"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "fd0e2a3c1fc04939a3fc1d0f61e09092"
server_key_iv: "cb7be2eb71a389819ab4534606e35fc6"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.760667 24724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "8b33505f8d6e417b9724193be517ae5a"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "fd0e2a3c1fc04939a3fc1d0f61e09092"
server_key_iv: "cb7be2eb71a389819ab4534606e35fc6"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.764681 24724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:43:55.767583 24739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.768926 24724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:43:55.769085 24724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "8b33505f8d6e417b9724193be517ae5a"
format_stamp: "Formatted at 2026-09-28 23:43:55 on dist-test-slave-c3s8"
server_key: "fd0e2a3c1fc04939a3fc1d0f61e09092"
server_key_iv: "cb7be2eb71a389819ab4534606e35fc6"
server_key_version: "encryptionkey@0"
I20260928 23:43:55.769217 24724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:55.782122 24724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:55.785763 24724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:55.785993 24724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:55.786628 24724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:55.787508 24724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:55.787580 24724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.787648 24724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:55.787700 24724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:55.797560 24724 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:46643
I20260928 23:43:55.797585 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:46643 every 8 connection(s)
I20260928 23:43:55.798533 24724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:43:55.800222 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24724
I20260928 23:43:55.800323 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:43:55.800631 21953 external_mini_cluster.cc:1428] Setting key d724001635ea631389d637254bcabab8
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639035, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:55.813052 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.801090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:53375 (local address 127.21.112.126:39347)
0928 23:43:55.801225 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:55.801230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:55.802007 (+ 777us) server_negotiation.cc:407] Connection header received
0928 23:43:55.803058 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:55.803060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:55.803105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:55.803175 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:55.804364 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.804906 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.806522 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.806750 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.806849 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.809298 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:55.809301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:55.809303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:55.809328 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:55.810857 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.811436 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.811439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.811440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.811534 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:55.811876 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:55.811881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:55.811883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:55.812067 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:55.812174 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:55.812717 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 23:43:55.812854 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:43:55.814051 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:55.801022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:39347 (local address 127.21.112.67:53375)
0928 23:43:55.801846 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:55.801895 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:55.802821 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:55.803331 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:55.803337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:55.803682 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:55.804203 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:55.804216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.805039 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.805041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:55.806397 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:55.806407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:55.806885 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:55.806887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:55.807051 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:55.807584 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:55.807600 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:55.809130 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:55.810997 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:55.811002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:55.811018 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:55.811315 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:55.811630 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:55.811634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:55.811636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:55.811780 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:55.812171 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:55.812179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:55.812454 (+ 275us) client_negotiation.cc:769] Sending connection context
0928 23:43:55.812665 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 23:43:55.813009 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":125,"threads_started":1}
I20260928 23:43:55.815366 24853 heartbeater.cc:347] Connected to a master server at 127.21.112.126:39347
I20260928 23:43:55.815586 24853 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:55.816015 24853 heartbeater.cc:511] Master 127.21.112.126:39347 requested a full tablet report, sending...
I20260928 23:43:55.816978 24391 ts_manager.cc:194] Registered new tserver with Master: 8b33505f8d6e417b9724193be517ae5a (127.21.112.67:46643)
I20260928 23:43:55.817593 24391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:53375
I20260928 23:43:55.826895 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:43:55.839660 24433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:43:55.838299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.21.112.126:39347)
0928 23:43:55.838435 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:55.838439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:55.838615 (+ 176us) server_negotiation.cc:407] Connection header received
0928 23:43:55.838727 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:55.838729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:55.838771 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:55.838849 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:55.839538 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49698: BlockingRecv error: recv got EOF from 127.0.0.1:49698 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:43:55.840610 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24456
I20260928 23:43:55.846333 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24590
I20260928 23:43:55.851537 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24724
I20260928 23:43:55.856637 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24362
2026-09-28T23:43:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2201 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24865](info): set up 2 sockets
Sep 28 23:43:55 dist-test-slave-c3s8 krb5kdc[24865](info): commencing operation
krb5kdc: starting...
W20260928 23:43:56.918259 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:43:57 dist-test-slave-c3s8 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:43:57.048992 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:42867
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:42867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:57.161463 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:57.161784 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:57.161827 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:57.165971 24878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:57.166040 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:57.166060 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:57.166077 24878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:57.166092 24878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:57.171574 24878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:42867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:57.172760 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:57.173875 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:57.182858 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:57.182871 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:57.182926 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:57.183033 24878 server_base.cc:1034] running on GCE node
I20260928 23:43:57.183765 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:57.184962 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:57.186151 24878 hybrid_clock.cc:648] HybridClock initialized: now 1790639037186130 us; error 31 us; skew 500 ppm
Sep 28 23:43:57 dist-test-slave-c3s8 krb5kdc[24865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639037, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:57.193606 24878 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:57.194639 24878 webserver.cc:460] Webserver started at http://127.21.112.126:42535/ using document root <none> and password file <none>
I20260928 23:43:57.195288 24878 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:57.195398 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:57.195622 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:57.197772 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "b498b970fb5a480eae92f7da6a69b291"
format_stamp: "Formatted at 2026-09-28 23:43:57 on dist-test-slave-c3s8"
server_key: "15a7f6ebe6eaa0f7a4d5d9fa5eb19f31"
server_key_iv: "f69bd6c4146743520d38eb7579e7643a"
server_key_version: "encryptionkey@0"
I20260928 23:43:57.198336 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "b498b970fb5a480eae92f7da6a69b291"
format_stamp: "Formatted at 2026-09-28 23:43:57 on dist-test-slave-c3s8"
server_key: "15a7f6ebe6eaa0f7a4d5d9fa5eb19f31"
server_key_iv: "f69bd6c4146743520d38eb7579e7643a"
server_key_version: "encryptionkey@0"
I20260928 23:43:57.202055 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:43:57.204391 24893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:57.205698 24878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:43:57.205835 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "b498b970fb5a480eae92f7da6a69b291"
format_stamp: "Formatted at 2026-09-28 23:43:57 on dist-test-slave-c3s8"
server_key: "15a7f6ebe6eaa0f7a4d5d9fa5eb19f31"
server_key_iv: "f69bd6c4146743520d38eb7579e7643a"
server_key_version: "encryptionkey@0"
I20260928 23:43:57.205946 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:57.234172 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:57.237272 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:57.237493 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:57.244937 24878 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:42867
I20260928 23:43:57.245398 24945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:42867 every 8 connection(s)
I20260928 23:43:57.246577 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:57.249796 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:57.255801 24946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291: Bootstrap starting.
I20260928 23:43:57.256313 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24878
I20260928 23:43:57.256453 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:57.256716 21953 external_mini_cluster.cc:1428] Setting key 3f8ddcc1ccc08add8efff3d0749bb51b
I20260928 23:43:57.258491 24946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:57.259366 24946 log.cc:824] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291: Log is configured to *not* fsync() on all Append() calls
I20260928 23:43:57.261767 24946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291: No bootstrap required, opened a new log
I20260928 23:43:57.266110 24946 raft_consensus.cc:348] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b498b970fb5a480eae92f7da6a69b291" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42867 } }
I20260928 23:43:57.266340 24946 raft_consensus.cc:374] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:57.266417 24946 raft_consensus.cc:729] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b498b970fb5a480eae92f7da6a69b291, State: Initialized, Role: FOLLOWER
Sep 28 23:43:57 dist-test-slave-c3s8 krb5kdc[24865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:57.266965 24946 consensus_queue.cc:260] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b498b970fb5a480eae92f7da6a69b291" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42867 } }
I20260928 23:43:57.267125 24946 raft_consensus.cc:388] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:57.267212 24946 raft_consensus.cc:482] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:57.267311 24946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:57.268638 24946 raft_consensus.cc:504] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b498b970fb5a480eae92f7da6a69b291" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42867 } }
I20260928 23:43:57.269085 24946 leader_election.cc:302] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b498b970fb5a480eae92f7da6a69b291; no voters:
I20260928 23:43:57.269482 24946 leader_election.cc:288] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:57.269644 24951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:57.269896 24951 raft_consensus.cc:686] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [term 1 LEADER]: Becoming Leader. State: Replica: b498b970fb5a480eae92f7da6a69b291, State: Running, Role: LEADER
I20260928 23:43:57.270251 24951 consensus_queue.cc:237] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b498b970fb5a480eae92f7da6a69b291" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42867 } }
I20260928 23:43:57.270957 24946 sys_catalog.cc:564] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:43:57.271978 24953 sys_catalog.cc:455] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b498b970fb5a480eae92f7da6a69b291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b498b970fb5a480eae92f7da6a69b291" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42867 } } }
I20260928 23:43:57.272106 24953 sys_catalog.cc:458] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:57.272503 24960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:57.273751 24952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b498b970fb5a480eae92f7da6a69b291. Latest consensus state: current_term: 1 leader_uuid: "b498b970fb5a480eae92f7da6a69b291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b498b970fb5a480eae92f7da6a69b291" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42867 } } }
I20260928 23:43:57.273835 24952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291 [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:57.274318 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:57.258066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.21.112.126:42867)
0928 23:43:57.258526 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:57.258548 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:57.258589 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:43:57.259488 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:57.259512 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:57.259929 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:57.260281 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:57.261176 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:57.262683 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:57.263675 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:57.264430 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:57.264636 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:57.267602 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:57.267608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:57.267628 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:57.267667 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:57.269947 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:57.270518 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:57.270524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:57.270531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:57.270677 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:57.270998 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:57.271001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:57.271003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:57.271547 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:57.271754 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:57.272951 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0928 23:43:57.273271 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
I20260928 23:43:57.275841 24960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:57.277649 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:43:57.281919 24960 catalog_manager.cc:1347] Generated new cluster ID: 76bdeb0d211642ab8e91ae5586f5e692
I20260928 23:43:57.282006 24960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:57.288471 24960 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:57.289366 24960 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:57.296118 24960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b498b970fb5a480eae92f7da6a69b291: Generated new TSK 0
I20260928 23:43:57.297318 24960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:43:57.297814 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:57.291939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.21.112.126:42867)
0928 23:43:57.292062 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:57.292067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:57.292386 (+ 319us) server_negotiation.cc:407] Connection header received
0928 23:43:57.292521 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:57.292524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:57.292566 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:57.292659 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:57.293268 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:57.294101 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:57.295368 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:57.295705 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:57.295830 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:57.296813 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:57.296817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:57.296820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:43:57.297267 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:57.297296 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:43:57.297312 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:57.297397 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:57.297613 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 23:43:57.297675 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:43:57.302416 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24878
2026-09-28T23:43:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1443 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:57 dist-test-slave-c3s8 krb5kdc[24979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:57 dist-test-slave-c3s8 krb5kdc[24979](info): set up 2 sockets
Sep 28 23:43:57 dist-test-slave-c3s8 krb5kdc[24979](info): commencing operation
krb5kdc: starting...
W20260928 23:43:58.418100 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:43:58 dist-test-slave-c3s8 krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:43:58.545114 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46601
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:58.657840 24992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:58.658196 24992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:58.658270 24992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:58.662608 24992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:43:58.662707 24992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:58.662747 24992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:58.662802 24992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:43:58.662842 24992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:43:58.668421 24992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:58.669684 24992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:58.670789 24992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:58.679734 24998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:58.679745 24997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:58.680040 24992 server_base.cc:1034] running on GCE node
W20260928 23:43:58.679736 25000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:58.680811 24992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:58.682046 24992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:58.683254 24992 hybrid_clock.cc:648] HybridClock initialized: now 1790639038683210 us; error 58 us; skew 500 ppm
Sep 28 23:43:58 dist-test-slave-c3s8 krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639038, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:58.686345 24992 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:43:58.687286 24992 webserver.cc:460] Webserver started at http://127.21.112.126:35729/ using document root <none> and password file <none>
I20260928 23:43:58.687902 24992 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:58.688011 24992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:58.688241 24992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:58.690622 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "399db97f1b0e4638a69d580851f020a4"
format_stamp: "Formatted at 2026-09-28 23:43:58 on dist-test-slave-c3s8"
server_key: "7cbfb2484957e51c8be64d627c54122e"
server_key_iv: "06af575933dcf78c489a9b9c75ee8636"
server_key_version: "encryptionkey@0"
I20260928 23:43:58.691263 24992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "399db97f1b0e4638a69d580851f020a4"
format_stamp: "Formatted at 2026-09-28 23:43:58 on dist-test-slave-c3s8"
server_key: "7cbfb2484957e51c8be64d627c54122e"
server_key_iv: "06af575933dcf78c489a9b9c75ee8636"
server_key_version: "encryptionkey@0"
I20260928 23:43:58.695164 24992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:43:58.697805 25007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:58.699101 24992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:43:58.699239 24992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "399db97f1b0e4638a69d580851f020a4"
format_stamp: "Formatted at 2026-09-28 23:43:58 on dist-test-slave-c3s8"
server_key: "7cbfb2484957e51c8be64d627c54122e"
server_key_iv: "06af575933dcf78c489a9b9c75ee8636"
server_key_version: "encryptionkey@0"
I20260928 23:43:58.699350 24992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:58.715596 24992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:58.719105 24992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:58.719317 24992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:58.726867 24992 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:46601
I20260928 23:43:58.726995 25059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:46601 every 8 connection(s)
I20260928 23:43:58.727882 24992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:43:58.730871 25060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:43:58.731580 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 24992
I20260928 23:43:58.731751 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:43:58.732071 21953 external_mini_cluster.cc:1428] Setting key 56959862637dcf36a1cc6748567e3804
I20260928 23:43:58.737849 25060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4: Bootstrap starting.
I20260928 23:43:58.740428 25060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4: Neither blocks nor log segments found. Creating new log.
I20260928 23:43:58.741460 25060 log.cc:824] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:43:58 dist-test-slave-c3s8 krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:58.744277 25060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4: No bootstrap required, opened a new log
I20260928 23:43:58.748973 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:58.733639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.21.112.126:46601)
0928 23:43:58.734080 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:58.734100 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:58.734147 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:43:58.734961 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:58.734988 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:58.735424 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:58.735803 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:58.736581 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:58.738061 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:58.739058 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:58.739749 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:58.739917 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:58.743136 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:58.743142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:58.743159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:58.743191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:58.745235 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:58.745840 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:58.745847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:58.745855 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:58.746005 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:58.746331 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:58.746334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:58.746336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:58.747044 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:58.747244 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:58.747564 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 23:43:58.747902 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
I20260928 23:43:58.749179 25060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db97f1b0e4638a69d580851f020a4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46601 } }
I20260928 23:43:58.749487 25060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:43:58.749577 25060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399db97f1b0e4638a69d580851f020a4, State: Initialized, Role: FOLLOWER
I20260928 23:43:58.750015 25060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db97f1b0e4638a69d580851f020a4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46601 } }
I20260928 23:43:58.750167 25060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:43:58.750233 25060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:43:58.750340 25060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:43:58.751289 25060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db97f1b0e4638a69d580851f020a4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46601 } }
I20260928 23:43:58.751660 25060 leader_election.cc:302] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 399db97f1b0e4638a69d580851f020a4; no voters:
I20260928 23:43:58.751982 25060 leader_election.cc:288] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:43:58.752252 25065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:43:58.752467 25065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [term 1 LEADER]: Becoming Leader. State: Replica: 399db97f1b0e4638a69d580851f020a4, State: Running, Role: LEADER
I20260928 23:43:58.752813 25065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db97f1b0e4638a69d580851f020a4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46601 } }
I20260928 23:43:58.753402 25060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:43:58.754583 25066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "399db97f1b0e4638a69d580851f020a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db97f1b0e4638a69d580851f020a4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46601 } } }
I20260928 23:43:58.754586 25067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 399db97f1b0e4638a69d580851f020a4. Latest consensus state: current_term: 1 leader_uuid: "399db97f1b0e4638a69d580851f020a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399db97f1b0e4638a69d580851f020a4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46601 } } }
I20260928 23:43:58.754750 25067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:58.754750 25066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:43:58.755203 25074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:43:58.758709 25074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:43:58.765059 25074 catalog_manager.cc:1347] Generated new cluster ID: e9853bd1296947fb8cfe84cee52e907f
I20260928 23:43:58.765154 25074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:43:58.800734 25074 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:43:58.801928 25074 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:43:58.812242 25074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 399db97f1b0e4638a69d580851f020a4: Generated new TSK 0
I20260928 23:43:58.812991 25074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:43:58.852774 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46601
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:58.968133 25086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:58.968487 25086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:58.968565 25086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:58.972848 25086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:58.972929 25086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:58.973021 25086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:43:58.978633 25086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:58.979786 25086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:58.980841 25086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:58.990845 25091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:58.990835 25094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:58.990829 25092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:58.991084 25086 server_base.cc:1034] running on GCE node
I20260928 23:43:58.991607 25086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:58.992261 25086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:58.993463 25086 hybrid_clock.cc:648] HybridClock initialized: now 1790639038993421 us; error 52 us; skew 500 ppm
Sep 28 23:43:58 dist-test-slave-c3s8 krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639038, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:58.996434 25086 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:43:58.997421 25086 webserver.cc:460] Webserver started at http://127.21.112.65:41537/ using document root <none> and password file <none>
I20260928 23:43:58.998039 25086 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:58.998106 25086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:58.998274 25086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:59.000434 25086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "96d48459b0b3424884efbd7e9c4ae4a3"
format_stamp: "Formatted at 2026-09-28 23:43:58 on dist-test-slave-c3s8"
server_key: "d6ee61fd9550d20d76afff92d4061853"
server_key_iv: "b5888518b244537bfe665c3d15d6df4b"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.001034 25086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "96d48459b0b3424884efbd7e9c4ae4a3"
format_stamp: "Formatted at 2026-09-28 23:43:58 on dist-test-slave-c3s8"
server_key: "d6ee61fd9550d20d76afff92d4061853"
server_key_iv: "b5888518b244537bfe665c3d15d6df4b"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.005174 25086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:43:59.007910 25101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.009315 25086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:43:59.009435 25086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "96d48459b0b3424884efbd7e9c4ae4a3"
format_stamp: "Formatted at 2026-09-28 23:43:58 on dist-test-slave-c3s8"
server_key: "d6ee61fd9550d20d76afff92d4061853"
server_key_iv: "b5888518b244537bfe665c3d15d6df4b"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.009595 25086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:59.034497 25086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:59.038097 25086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:59.038259 25086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:59.038801 25086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:59.039754 25086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:59.039801 25086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.039840 25086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:59.039891 25086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.049389 25086 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:37729
I20260928 23:43:59.049430 25214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:37729 every 8 connection(s)
I20260928 23:43:59.050330 25086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:43:59.059990 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25086
I20260928 23:43:59.060135 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:43:59.060464 21953 external_mini_cluster.cc:1428] Setting key fcc44bd7bf7af8275c85d5b8fe2c3279
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639038, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:59.065666 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.052322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:42767 (local address 127.21.112.126:46601)
0928 23:43:59.052482 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:59.052485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:59.053438 (+ 953us) server_negotiation.cc:407] Connection header received
0928 23:43:59.055003 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:59.055006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:59.055056 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:59.055144 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:59.056552 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.057162 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.058430 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.058723 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.058813 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.061701 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:59.061704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:59.061706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:59.061731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:59.063532 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.064082 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.064085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.064087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.064173 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.064535 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.064538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.064540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.064728 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:59.064833 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:59.065343 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 23:43:59.065478 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:43:59.066567 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.052745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46601 (local address 127.21.112.65:42767)
0928 23:43:59.053274 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:59.053330 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:59.054745 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:59.055400 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:59.055408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:59.055795 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:59.056353 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:59.056367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.057294 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.057297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:59.058298 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:59.058308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.058882 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.058885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:59.059038 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.059709 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:59.059726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:59.061457 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:59.063666 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:59.063672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:59.063689 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:59.063970 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:59.064271 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:59.064274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:59.064277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:59.064435 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:59.064832 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:59.064840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:59.065103 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 23:43:59.065290 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 23:43:59.065675 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":164,"threads_started":1}
I20260928 23:43:59.068039 25215 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46601
I20260928 23:43:59.068296 25215 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:59.068823 25215 heartbeater.cc:511] Master 127.21.112.126:46601 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:43:59.070644 25024 ts_manager.cc:194] Registered new tserver with Master: 96d48459b0b3424884efbd7e9c4ae4a3 (127.21.112.65:37729)
I20260928 23:43:59.071839 25024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:42767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:43:59.089139 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46601
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:59.204314 25220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:59.204654 25220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:59.204733 25220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:59.209286 25220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:59.209409 25220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:59.209524 25220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:43:59.215477 25220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:59.216599 25220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:59.217684 25220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:59.228084 25225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:59.228219 25220 server_base.cc:1034] running on GCE node
W20260928 23:43:59.228084 25228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:59.228111 25226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:59.228892 25220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:59.229691 25220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:59.230881 25220 hybrid_clock.cc:648] HybridClock initialized: now 1790639039230888 us; error 71 us; skew 500 ppm
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639039, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:59.233927 25220 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:43:59.234845 25220 webserver.cc:460] Webserver started at http://127.21.112.66:35035/ using document root <none> and password file <none>
I20260928 23:43:59.235405 25220 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:59.235468 25220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:59.235649 25220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:59.237879 25220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "0ba878d1d4fa459f853836f8812e52dc"
format_stamp: "Formatted at 2026-09-28 23:43:59 on dist-test-slave-c3s8"
server_key: "26d48d5a689c54d1864e21b475033088"
server_key_iv: "ebae501ef14248434792716c16f30104"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.238493 25220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "0ba878d1d4fa459f853836f8812e52dc"
format_stamp: "Formatted at 2026-09-28 23:43:59 on dist-test-slave-c3s8"
server_key: "26d48d5a689c54d1864e21b475033088"
server_key_iv: "ebae501ef14248434792716c16f30104"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.242566 25220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 23:43:59.245260 25235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.246901 25220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:43:59.247054 25220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "0ba878d1d4fa459f853836f8812e52dc"
format_stamp: "Formatted at 2026-09-28 23:43:59 on dist-test-slave-c3s8"
server_key: "26d48d5a689c54d1864e21b475033088"
server_key_iv: "ebae501ef14248434792716c16f30104"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.247175 25220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:59.260164 25220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:59.263785 25220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:59.263953 25220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:59.264495 25220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:59.265347 25220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:59.265394 25220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.265439 25220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:59.265460 25220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.274549 25220 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:41133
I20260928 23:43:59.274569 25348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:41133 every 8 connection(s)
I20260928 23:43:59.275529 25220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639039, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:59.285782 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25220
I20260928 23:43:59.285897 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:43:59.286170 21953 external_mini_cluster.cc:1428] Setting key 0cfea77042b67efbac640b9e5f291aa2
I20260928 23:43:59.290215 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.277499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:44285 (local address 127.21.112.126:46601)
0928 23:43:59.277670 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:59.277673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:59.278598 (+ 925us) server_negotiation.cc:407] Connection header received
0928 23:43:59.279989 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:59.279993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:59.280055 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:59.280157 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:59.281392 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.282010 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.283149 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.283381 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.283488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.286133 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:59.286136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:59.286138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:59.286163 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:59.287808 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.288422 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.288426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.288428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.288510 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.288876 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.288879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.288881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.289159 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:59.289282 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:59.289855 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 23:43:59.290017 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 23:43:59.290989 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.277963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46601 (local address 127.21.112.66:44285)
0928 23:43:59.278445 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:59.278491 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:59.279765 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:59.280338 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:59.280346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:59.280729 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:59.281241 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:59.281254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.282141 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.282145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:59.283026 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:59.283034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.283532 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.283536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:59.283705 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.284292 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:59.284308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:59.285953 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:59.287988 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:59.287994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:59.288030 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:59.288303 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:59.288606 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:59.288609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:59.288611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:59.288757 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:59.289280 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:59.289285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:59.289594 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 23:43:59.289786 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 23:43:59.290077 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260928 23:43:59.292173 25349 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46601
I20260928 23:43:59.292460 25349 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:59.293174 25349 heartbeater.cc:511] Master 127.21.112.126:46601 requested a full tablet report, sending...
I20260928 23:43:59.294476 25024 ts_manager.cc:194] Registered new tserver with Master: 0ba878d1d4fa459f853836f8812e52dc (127.21.112.66:41133)
I20260928 23:43:59.295089 25024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:44285
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:43:59.314740 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46601
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:43:59.426512 25354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:43:59.426756 25354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:43:59.426798 25354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:43:59.430836 25354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:43:59.430899 25354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:43:59.430958 25354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:43:59.436299 25354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:43:59.437366 25354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:43:59.438417 25354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:43:59.448698 25360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:43:59.448688 25359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:59.448841 25354 server_base.cc:1034] running on GCE node
W20260928 23:43:59.448688 25362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:43:59.449481 25354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:43:59.450212 25354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:43:59.451375 25354 hybrid_clock.cc:648] HybridClock initialized: now 1790639039451349 us; error 42 us; skew 500 ppm
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639039, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:43:59.454383 25354 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:43:59.455381 25354 webserver.cc:460] Webserver started at http://127.21.112.67:34707/ using document root <none> and password file <none>
I20260928 23:43:59.455950 25354 fs_manager.cc:359] Metadata directory not provided
I20260928 23:43:59.456017 25354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:43:59.456243 25354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:43:59.458434 25354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "2d0b238e32e34779a3161006abd1b5ac"
format_stamp: "Formatted at 2026-09-28 23:43:59 on dist-test-slave-c3s8"
server_key: "b066ed160b6ed91903d6b5331b0af415"
server_key_iv: "6d4739c17432856ebaad079951ec40bd"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.459002 25354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "2d0b238e32e34779a3161006abd1b5ac"
format_stamp: "Formatted at 2026-09-28 23:43:59 on dist-test-slave-c3s8"
server_key: "b066ed160b6ed91903d6b5331b0af415"
server_key_iv: "6d4739c17432856ebaad079951ec40bd"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.463155 25354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:43:59.465927 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.467413 25354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:43:59.467566 25354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "2d0b238e32e34779a3161006abd1b5ac"
format_stamp: "Formatted at 2026-09-28 23:43:59 on dist-test-slave-c3s8"
server_key: "b066ed160b6ed91903d6b5331b0af415"
server_key_iv: "6d4739c17432856ebaad079951ec40bd"
server_key_version: "encryptionkey@0"
I20260928 23:43:59.467690 25354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:43:59.479418 25354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:43:59.483229 25354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:43:59.483456 25354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:43:59.484079 25354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:43:59.484974 25354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:43:59.485041 25354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.485108 25354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:43:59.485157 25354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:43:59.494383 25354 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:37939
I20260928 23:43:59.494410 25482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:37939 every 8 connection(s)
I20260928 23:43:59.495275 25354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:43:59.501360 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25354
I20260928 23:43:59.501505 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:43:59.501829 21953 external_mini_cluster.cc:1428] Setting key 9a4cc73c2144f33329fc9f193120de3f
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639039, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:59.510046 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.497208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:50103 (local address 127.21.112.126:46601)
0928 23:43:59.497356 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:59.497360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:59.498329 (+ 969us) server_negotiation.cc:407] Connection header received
0928 23:43:59.499750 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:59.499753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:59.499811 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:59.499912 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:59.501074 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.501744 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.503298 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.503634 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.503743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.506332 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:59.506335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:59.506337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:59.506359 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:59.507738 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.508400 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.508406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.508409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.508505 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.508908 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.508915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.508917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.509164 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:59.509314 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:59.509824 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 23:43:59.509931 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:43:59.510883 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.497656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46601 (local address 127.21.112.67:50103)
0928 23:43:59.498170 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 23:43:59.498213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:43:59.499522 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:43:59.500086 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:43:59.500093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:43:59.500452 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:43:59.500937 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:59.500950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.501876 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.501879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:59.503161 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:43:59.503172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.503768 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.503770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:43:59.504022 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.504665 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:43:59.504681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:43:59.506183 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:43:59.507849 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:59.507854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:59.507866 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:59.508302 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:59.508597 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:43:59.508600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:43:59.508620 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:43:59.508792 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:43:59.509301 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:43:59.509306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:43:59.509583 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 23:43:59.509769 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 23:43:59.510038 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260928 23:43:59.512049 25483 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46601
I20260928 23:43:59.512274 25483 heartbeater.cc:464] Registering TS with master...
I20260928 23:43:59.512809 25483 heartbeater.cc:511] Master 127.21.112.126:46601 requested a full tablet report, sending...
I20260928 23:43:59.513916 25024 ts_manager.cc:194] Registered new tserver with Master: 2d0b238e32e34779a3161006abd1b5ac (127.21.112.67:37939)
I20260928 23:43:59.514669 25024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:50103
I20260928 23:43:59.515918 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639039, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[24979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639039, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:43:59.542303 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:43:59.533019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.21.112.126:46601)
0928 23:43:59.533172 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:43:59.533176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:43:59.533401 (+ 225us) server_negotiation.cc:407] Connection header received
0928 23:43:59.533601 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:43:59.533604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:43:59.533651 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:43:59.533719 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:43:59.534441 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.535049 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.535980 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:43:59.536185 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:43:59.536265 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:43:59.538960 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:43:59.538963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:43:59.538965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:43:59.539014 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:43:59.540501 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.541037 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.541040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.541041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.541122 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:43:59.541365 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:43:59.541372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:43:59.541374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:43:59.541662 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:43:59.541772 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:43:59.542066 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 23:43:59.542175 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 23:43:59.542757 25024 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56746
I20260928 23:43:59.543679 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25086
I20260928 23:43:59.549844 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25220
I20260928 23:43:59.556059 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25354
I20260928 23:43:59.561910 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 24992
2026-09-28T23:43:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2263 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[25495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[25495](info): set up 2 sockets
Sep 28 23:43:59 dist-test-slave-c3s8 krb5kdc[25495](info): commencing operation
krb5kdc: starting...
W20260928 23:44:00.666433 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:00 dist-test-slave-c3s8 krb5kdc[25495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:44:00.801270 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43169
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:00.915012 25508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:00.915372 25508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:00.915457 25508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:00.919526 25508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:00.919618 25508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:00.919659 25508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:00.919693 25508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:00.919726 25508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:00.925225 25508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:00.926367 25508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:00.927412 25508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:00.936364 25514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:00.936383 25513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:00.936481 25508 server_base.cc:1034] running on GCE node
W20260928 23:44:00.936383 25516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:00.937165 25508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:00.938491 25508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:00.939689 25508 hybrid_clock.cc:648] HybridClock initialized: now 1790639040939673 us; error 31 us; skew 500 ppm
Sep 28 23:44:00 dist-test-slave-c3s8 krb5kdc[25495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639040, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:00.942732 25508 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:44:00.943714 25508 webserver.cc:460] Webserver started at http://127.21.112.126:46479/ using document root <none> and password file <none>
I20260928 23:44:00.944306 25508 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:00.944371 25508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:00.944582 25508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:00.946813 25508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "210be5f55c9045238d44efcc260cc5f1"
format_stamp: "Formatted at 2026-09-28 23:44:00 on dist-test-slave-c3s8"
server_key: "a317e0cfdc2c175609523c16e7b23342"
server_key_iv: "4433cb25004386fc3f5cc1c97874c73b"
server_key_version: "encryptionkey@0"
I20260928 23:44:00.947422 25508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "210be5f55c9045238d44efcc260cc5f1"
format_stamp: "Formatted at 2026-09-28 23:44:00 on dist-test-slave-c3s8"
server_key: "a317e0cfdc2c175609523c16e7b23342"
server_key_iv: "4433cb25004386fc3f5cc1c97874c73b"
server_key_version: "encryptionkey@0"
I20260928 23:44:00.951601 25508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:00.954315 25523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:00.955732 25508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:00.955888 25508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "210be5f55c9045238d44efcc260cc5f1"
format_stamp: "Formatted at 2026-09-28 23:44:00 on dist-test-slave-c3s8"
server_key: "a317e0cfdc2c175609523c16e7b23342"
server_key_iv: "4433cb25004386fc3f5cc1c97874c73b"
server_key_version: "encryptionkey@0"
I20260928 23:44:00.956079 25508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:00.974700 25508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:00.978361 25508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:00.978585 25508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:00.986677 25508 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:43169
I20260928 23:44:00.986708 25575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:43169 every 8 connection(s)
I20260928 23:44:00.987686 25508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:00.988049 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25508
I20260928 23:44:00.988191 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:00.988512 21953 external_mini_cluster.cc:1428] Setting key 893dcae5f6063d7c2378163ccd981968
I20260928 23:44:00.990986 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:00.996955 25576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1: Bootstrap starting.
Sep 28 23:44:00 dist-test-slave-c3s8 krb5kdc[25495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:00.999603 25576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:01.000370 25576 log.cc:824] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:01.002632 25576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1: No bootstrap required, opened a new log
I20260928 23:44:01.002979 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:00.989990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.21.112.126:43169)
0928 23:44:00.990456 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:00.990474 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:00.990511 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:44:00.991248 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:00.991267 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:00.991618 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:00.992022 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:00.992839 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:00.993925 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:00.994771 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:00.995256 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:00.995428 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:00.998074 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:00.998079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:00.998093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:00.998124 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:00.999859 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.000297 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.000303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.000309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.000434 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.000710 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.000714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.000715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.001309 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:01.001510 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:01.001835 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 23:44:01.002156 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":152,"threads_started":1}
I20260928 23:44:01.006753 25576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210be5f55c9045238d44efcc260cc5f1" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43169 } }
I20260928 23:44:01.007031 25576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:01.007121 25576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210be5f55c9045238d44efcc260cc5f1, State: Initialized, Role: FOLLOWER
I20260928 23:44:01.007639 25576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210be5f55c9045238d44efcc260cc5f1" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43169 } }
I20260928 23:44:01.007798 25576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:01.007884 25576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:01.007994 25576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:01.009001 25576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210be5f55c9045238d44efcc260cc5f1" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43169 } }
I20260928 23:44:01.009389 25576 leader_election.cc:302] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210be5f55c9045238d44efcc260cc5f1; no voters:
I20260928 23:44:01.009779 25576 leader_election.cc:288] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:01.009922 25581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:01.010179 25581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [term 1 LEADER]: Becoming Leader. State: Replica: 210be5f55c9045238d44efcc260cc5f1, State: Running, Role: LEADER
I20260928 23:44:01.010509 25581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210be5f55c9045238d44efcc260cc5f1" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43169 } }
I20260928 23:44:01.011138 25576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:01.012117 25583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210be5f55c9045238d44efcc260cc5f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210be5f55c9045238d44efcc260cc5f1" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43169 } } }
I20260928 23:44:01.012269 25583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:01.012941 25582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210be5f55c9045238d44efcc260cc5f1. Latest consensus state: current_term: 1 leader_uuid: "210be5f55c9045238d44efcc260cc5f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210be5f55c9045238d44efcc260cc5f1" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43169 } } }
I20260928 23:44:01.013065 25582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:01.013896 25590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:01.016600 25590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:01.022393 25590 catalog_manager.cc:1347] Generated new cluster ID: 72e78382ae12439b818f41c3594fdd9d
I20260928 23:44:01.022487 25590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:01.031857 25590 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:01.032912 25590 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:01.039791 25590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210be5f55c9045238d44efcc260cc5f1: Generated new TSK 0
I20260928 23:44:01.040560 25590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:44:01.081719 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43169
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:01.195499 25602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:01.195820 25602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:01.195866 25602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:01.199972 25602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:01.200034 25602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:01.200094 25602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:01.205760 25602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:01.206874 25602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:01.207950 25602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:01.218323 25608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:01.218384 25602 server_base.cc:1034] running on GCE node
W20260928 23:44:01.218334 25607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:01.218407 25610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:01.219030 25602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:01.219642 25602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:01.220803 25602 hybrid_clock.cc:648] HybridClock initialized: now 1790639041220784 us; error 31 us; skew 500 ppm
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:01.223841 25602 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:44:01.224736 25602 webserver.cc:460] Webserver started at http://127.21.112.65:40449/ using document root <none> and password file <none>
I20260928 23:44:01.225296 25602 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:01.225390 25602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:01.225674 25602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:01.227752 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "b1f5839031c4470d8857cb54d673aeca"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "ab21845c5473fae8c47563f592f94471"
server_key_iv: "ba42602672f50c56ffda0b7127e71beb"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.228297 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "b1f5839031c4470d8857cb54d673aeca"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "ab21845c5473fae8c47563f592f94471"
server_key_iv: "ba42602672f50c56ffda0b7127e71beb"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.232676 25602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:44:01.235374 25617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.236814 25602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:01.236979 25602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "b1f5839031c4470d8857cb54d673aeca"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "ab21845c5473fae8c47563f592f94471"
server_key_iv: "ba42602672f50c56ffda0b7127e71beb"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.237113 25602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:01.253330 25602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:01.256844 25602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:01.257071 25602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:01.257757 25602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:01.258658 25602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:01.258728 25602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.258797 25602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:01.258843 25602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.268452 25602 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:40275
I20260928 23:44:01.268502 25730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:40275 every 8 connection(s)
I20260928 23:44:01.269425 25602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:01.278662 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25602
I20260928 23:44:01.278810 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:01.279141 21953 external_mini_cluster.cc:1428] Setting key 810bae767e59d0c2ee5f49dfb8d36e5b
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:01.284536 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.271388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:50255 (local address 127.21.112.126:43169)
0928 23:44:01.271534 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:01.271537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:01.272429 (+ 892us) server_negotiation.cc:407] Connection header received
0928 23:44:01.273960 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:01.273964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:01.274013 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:01.274098 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:01.275351 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.275896 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.277158 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.277429 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.277525 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.280644 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:01.280653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:01.280655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:01.280680 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:01.282303 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.282864 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.282869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.282871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.282958 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.283333 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.283336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.283338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.283534 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:01.283642 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:01.284202 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 23:44:01.284368 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:44:01.285460 25733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.271811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43169 (local address 127.21.112.65:50255)
0928 23:44:01.272257 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:01.272302 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:01.273750 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:01.274269 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:01.274276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:01.274642 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:01.275196 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:01.275210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.276073 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.276076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:01.277017 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:01.277027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.277642 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.277646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:01.277808 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.278449 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:01.278471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:01.280458 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:01.282436 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:01.282443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:01.282461 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:01.282754 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:01.283049 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:01.283054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:01.283056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:01.283214 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:01.283641 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:01.283650 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:01.283939 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 23:44:01.284164 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 23:44:01.284537 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260928 23:44:01.286881 25731 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43169
I20260928 23:44:01.287196 25731 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:01.287863 25731 heartbeater.cc:511] Master 127.21.112.126:43169 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:44:01.289716 25540 ts_manager.cc:194] Registered new tserver with Master: b1f5839031c4470d8857cb54d673aeca (127.21.112.65:40275)
I20260928 23:44:01.290904 25540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:50255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:44:01.309023 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43169
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:01.423734 25736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:01.424013 25736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:01.424083 25736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:01.428290 25736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:01.428356 25736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:01.428437 25736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:01.434022 25736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:01.435230 25736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:01.436290 25736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:01.446415 25742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:01.446415 25741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:01.446442 25744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:01.446926 25736 server_base.cc:1034] running on GCE node
I20260928 23:44:01.447559 25736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:01.448253 25736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:01.449441 25736 hybrid_clock.cc:648] HybridClock initialized: now 1790639041449425 us; error 36 us; skew 500 ppm
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:01.452327 25736 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:44:01.453280 25736 webserver.cc:460] Webserver started at http://127.21.112.66:38273/ using document root <none> and password file <none>
I20260928 23:44:01.453910 25736 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:01.453985 25736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:01.454161 25736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:01.456301 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "27850498a873457e92d55d4354236267"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "3659197bbfd235be201c3560d1f302c2"
server_key_iv: "4e2e877ceebadad6be5955cfe86d1087"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.456840 25736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "27850498a873457e92d55d4354236267"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "3659197bbfd235be201c3560d1f302c2"
server_key_iv: "4e2e877ceebadad6be5955cfe86d1087"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.460843 25736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:44:01.463516 25751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.464818 25736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:44:01.464927 25736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "27850498a873457e92d55d4354236267"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "3659197bbfd235be201c3560d1f302c2"
server_key_iv: "4e2e877ceebadad6be5955cfe86d1087"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.465027 25736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:01.487987 25736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:01.491539 25736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:01.491706 25736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:01.492283 25736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:01.493139 25736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:01.493185 25736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.493228 25736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:01.493243 25736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.501861 25736 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:43915
I20260928 23:44:01.501904 25864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:43915 every 8 connection(s)
I20260928 23:44:01.502951 25736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:01.506430 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25736
I20260928 23:44:01.506552 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:01.506814 21953 external_mini_cluster.cc:1428] Setting key 1c73335195f81f940a361f4afbd928e8
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:44:01.518452 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.504717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:56577 (local address 127.21.112.126:43169)
0928 23:44:01.504844 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:01.504848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:01.505724 (+ 876us) server_negotiation.cc:407] Connection header received
0928 23:44:01.507328 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:01.507331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:01.507389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:01.507488 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:01.508653 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.509173 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.510932 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.511167 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.511279 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.514623 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:01.514632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:01.514634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:01.514678 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:01.516157 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.516736 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.516740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.516741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.516817 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.517171 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.517174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.517176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.517399 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:01.517505 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:01.518076 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 23:44:01.518239 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:44:01.519474 25867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.505066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43169 (local address 127.21.112.66:56577)
0928 23:44:01.505518 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:01.505616 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:01.507118 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:01.507666 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:01.507673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:01.508035 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:01.508515 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:01.508528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.509323 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.509326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:01.510795 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:01.510809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.511355 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.511362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:01.511612 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.512323 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:01.512350 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:01.514406 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:01.516284 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:01.516291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:01.516309 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:01.516615 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:01.516915 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:01.516918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:01.516920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:01.517072 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:01.517503 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:01.517521 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:01.517820 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 23:44:01.518015 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 23:44:01.518372 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260928 23:44:01.520721 25865 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43169
I20260928 23:44:01.520987 25865 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:01.521488 25865 heartbeater.cc:511] Master 127.21.112.126:43169 requested a full tablet report, sending...
I20260928 23:44:01.522605 25540 ts_manager.cc:194] Registered new tserver with Master: 27850498a873457e92d55d4354236267 (127.21.112.66:43915)
I20260928 23:44:01.523227 25540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:56577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:44:01.536957 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43169
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:01.656993 25870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:01.657342 25870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:01.657423 25870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:01.661859 25870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:01.661954 25870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:01.662050 25870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:01.667603 25870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:01.668867 25870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:01.670054 25870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:01.680545 25876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:01.680547 25878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:01.680642 25870 server_base.cc:1034] running on GCE node
W20260928 23:44:01.680547 25875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:01.681370 25870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:01.682209 25870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:01.683445 25870 hybrid_clock.cc:648] HybridClock initialized: now 1790639041683400 us; error 59 us; skew 500 ppm
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:01.686607 25870 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:44:01.687634 25870 webserver.cc:460] Webserver started at http://127.21.112.67:42915/ using document root <none> and password file <none>
I20260928 23:44:01.688236 25870 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:01.688324 25870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:01.688560 25870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:01.690793 25870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "d70b26d56d374c05ba4700e2ff51930b"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "1ea0b28eb50b66e7934089db5d2f8898"
server_key_iv: "a5f5c1dc9df495a8825873a919cb8cdd"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.691375 25870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "d70b26d56d374c05ba4700e2ff51930b"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "1ea0b28eb50b66e7934089db5d2f8898"
server_key_iv: "a5f5c1dc9df495a8825873a919cb8cdd"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.695423 25870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:01.698136 25885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.699575 25870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:01.699729 25870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "d70b26d56d374c05ba4700e2ff51930b"
format_stamp: "Formatted at 2026-09-28 23:44:01 on dist-test-slave-c3s8"
server_key: "1ea0b28eb50b66e7934089db5d2f8898"
server_key_iv: "a5f5c1dc9df495a8825873a919cb8cdd"
server_key_version: "encryptionkey@0"
I20260928 23:44:01.699855 25870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:01.720674 25870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:01.724205 25870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:01.724434 25870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:01.725076 25870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:01.725991 25870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:01.726059 25870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.726122 25870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:01.726162 25870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:01.735500 25870 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:34821
I20260928 23:44:01.735520 25998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:34821 every 8 connection(s)
I20260928 23:44:01.736495 25870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:01.744007 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 25870
I20260928 23:44:01.744148 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:01.744475 21953 external_mini_cluster.cc:1428] Setting key 348a98a49f214ccdb96aa3f17705a2b2
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:01.752025 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.738432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:56471 (local address 127.21.112.126:43169)
0928 23:44:01.738619 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:01.738622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:01.739447 (+ 825us) server_negotiation.cc:407] Connection header received
0928 23:44:01.740942 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:01.740946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:01.741053 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:01.741192 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:01.742594 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.743126 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.744739 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.745131 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.745230 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.748314 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:01.748317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:01.748319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:01.748347 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:01.749860 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.750441 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.750443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.750445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.750523 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.750907 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.750909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.750911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.751096 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:01.751172 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:01.751786 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 23:44:01.751898 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:44:01.753019 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.738842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43169 (local address 127.21.112.67:56471)
0928 23:44:01.739289 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:01.739334 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:01.740717 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:01.741431 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:01.741439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:01.741924 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:01.742433 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:01.742459 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.743249 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.743252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:01.744578 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:01.744591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.745316 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.745326 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:01.745576 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.746309 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:01.746331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:01.748150 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:01.750013 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:01.750018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:01.750033 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:01.750318 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:01.750645 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:01.750648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:01.750649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:01.750813 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:01.751192 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:01.751198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:01.751559 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 23:44:01.751735 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 23:44:01.752052 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260928 23:44:01.754400 25999 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43169
I20260928 23:44:01.754639 25999 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:01.755159 25999 heartbeater.cc:511] Master 127.21.112.126:43169 requested a full tablet report, sending...
I20260928 23:44:01.756299 25540 ts_manager.cc:194] Registered new tserver with Master: d70b26d56d374c05ba4700e2ff51930b (127.21.112.67:34821)
I20260928 23:44:01.757028 25540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:56471
I20260928 23:44:01.759107 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:44:01.787277 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.776821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.21.112.65:40275)
0928 23:44:01.777123 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:01.777141 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:01.777364 (+ 223us) server_negotiation.cc:407] Connection header received
0928 23:44:01.777432 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:01.777437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:01.777735 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:01.777877 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:01.778609 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.779450 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.780297 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.780707 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.780798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.783538 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:01.783548 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:01.783575 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:01.783604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:01.785344 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.785862 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.785868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.785869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.785983 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.786333 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.786335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.786336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.786611 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:01.786770 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:01.786999 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 23:44:01.787139 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":118,"threads_started":1}
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[25495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:01.799341 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:01.790386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.21.112.126:43169)
0928 23:44:01.790551 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:01.790555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:01.790570 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:01.790704 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:01.790706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:01.790787 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:01.790868 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:01.791524 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.792104 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.792912 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:01.793163 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:01.793251 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:01.796145 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:01.796148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:01.796151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:01.796177 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:01.797455 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.797975 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.797979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.797980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.798081 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:01.798342 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:01.798344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:01.798345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:01.798622 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:01.798730 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:01.798990 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 23:44:01.799148 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 23:44:01.799791 25540 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55554
I20260928 23:44:01.800479 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25602
I20260928 23:44:01.806550 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25736
I20260928 23:44:01.812247 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25870
I20260928 23:44:01.817413 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 25508
2026-09-28T23:44:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2253 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[26011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[26011](info): set up 2 sockets
Sep 28 23:44:01 dist-test-slave-c3s8 krb5kdc[26011](info): commencing operation
krb5kdc: starting...
W20260928 23:44:02.870118 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:02 dist-test-slave-c3s8 krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:44:03.005399 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:35331
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:38583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:03.119557 26024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:03.119913 26024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:03.119997 26024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:03.124238 26024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:03.124336 26024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:03.124374 26024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:03.124430 26024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:03.124472 26024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:03.129940 26024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:35331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:03.131129 26024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:03.132239 26024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:03.141175 26032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:03.141270 26030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:03.141270 26029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:03.141767 26024 server_base.cc:1034] running on GCE node
I20260928 23:44:03.142400 26024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:03.143567 26024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:03.144771 26024 hybrid_clock.cc:648] HybridClock initialized: now 1790639043144722 us; error 65 us; skew 500 ppm
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:03.162470 26024 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:44:03.163426 26024 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:03.163534 26024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:03.163784 26024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:03.166154 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "298f3c0dd3f5445d9a9e9a44282d9248"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "7e169c6db6ddd6860376f6ee25579b17"
server_key_iv: "423362a665c5f4571c2d6a3431361e72"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.166803 26024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "298f3c0dd3f5445d9a9e9a44282d9248"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "7e169c6db6ddd6860376f6ee25579b17"
server_key_iv: "423362a665c5f4571c2d6a3431361e72"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.170971 26024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:03.173705 26038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.175096 26024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:03.175266 26024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "298f3c0dd3f5445d9a9e9a44282d9248"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "7e169c6db6ddd6860376f6ee25579b17"
server_key_iv: "423362a665c5f4571c2d6a3431361e72"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.175395 26024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:03.220698 26024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:03.224174 26024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:03.224380 26024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:03.231493 26024 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:35331
I20260928 23:44:03.231812 26090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:35331 every 8 connection(s)
I20260928 23:44:03.232185 26024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:03.232975 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26024
I20260928 23:44:03.233116 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:03.233392 21953 external_mini_cluster.cc:1428] Setting key 543cb6479cf7fcac295cdcc40f7db13d
I20260928 23:44:03.235502 26091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:03.242631 26091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: Bootstrap starting.
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:03.244972 26091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:03.245771 26091 log.cc:824] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:03.247812 26091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: No bootstrap required, opened a new log
I20260928 23:44:03.248392 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:03.234795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.21.112.126:35331)
0928 23:44:03.235212 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:03.235228 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:03.235264 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:44:03.235995 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:03.236014 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:03.236370 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:03.236678 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:03.237650 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.238704 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.239889 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.240394 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.240531 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:03.243560 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:03.243565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:03.243580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:03.243608 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:03.245440 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:03.245924 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:03.245929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:03.245934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:03.246043 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:03.246324 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:03.246327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:03.246329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:03.246735 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:03.246932 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:03.247243 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 23:44:03.247552 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":144,"threads_started":1}
I20260928 23:44:03.251735 26091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 35331 } }
I20260928 23:44:03.252007 26091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:03.252103 26091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298f3c0dd3f5445d9a9e9a44282d9248, State: Initialized, Role: FOLLOWER
I20260928 23:44:03.252537 26091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 35331 } }
I20260928 23:44:03.252692 26091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:03.252774 26091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:03.252893 26091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:03.254017 26091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 35331 } }
I20260928 23:44:03.254434 26091 leader_election.cc:302] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298f3c0dd3f5445d9a9e9a44282d9248; no voters:
I20260928 23:44:03.254817 26091 leader_election.cc:288] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:03.254956 26096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:03.255256 26096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [term 1 LEADER]: Becoming Leader. State: Replica: 298f3c0dd3f5445d9a9e9a44282d9248, State: Running, Role: LEADER
I20260928 23:44:03.255646 26096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 35331 } }
I20260928 23:44:03.256220 26091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:03.257611 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 35331 } } }
I20260928 23:44:03.257754 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:03.257728 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298f3c0dd3f5445d9a9e9a44282d9248. Latest consensus state: current_term: 1 leader_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f3c0dd3f5445d9a9e9a44282d9248" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 35331 } } }
I20260928 23:44:03.257915 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248 [sys.catalog]: This master's current role is: LEADER
W20260928 23:44:03.261584 26111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:44:03.261668 26111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:44:03.261739 26105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:03.262452 26105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:03.269124 26105 catalog_manager.cc:1347] Generated new cluster ID: 37cbbadb86fc4a1fa5134f97066b0f4c
I20260928 23:44:03.269212 26105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:03.292270 26105 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:03.293327 26105 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:03.299288 26105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: Generated new TSK 0
I20260928 23:44:03.300168 26105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:44:03.341380 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:35331
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:38583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:03.458505 26117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:03.458786 26117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:03.458860 26117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:03.462975 26117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:03.463044 26117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:03.463145 26117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:03.468696 26117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:03.469899 26117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:03.470973 26117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:03.481176 26122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:03.481176 26123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:03.481319 26117 server_base.cc:1034] running on GCE node
W20260928 23:44:03.481570 26125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:03.481976 26117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:03.482575 26117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:03.483740 26117 hybrid_clock.cc:648] HybridClock initialized: now 1790639043483704 us; error 51 us; skew 500 ppm
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:03.486799 26117 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:44:03.487622 26117 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:03.487710 26117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:03.487941 26117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:03.490135 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "226633ba66a04650a369247d072292a9"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "87776ee8ec9f27b80e8819c71250a76a"
server_key_iv: "c1d924ccde5f06b667f33960d677abb4"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.490749 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "226633ba66a04650a369247d072292a9"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "87776ee8ec9f27b80e8819c71250a76a"
server_key_iv: "c1d924ccde5f06b667f33960d677abb4"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.494550 26117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:03.497035 26131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.498340 26117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:03.498472 26117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "226633ba66a04650a369247d072292a9"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "87776ee8ec9f27b80e8819c71250a76a"
server_key_iv: "c1d924ccde5f06b667f33960d677abb4"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.498580 26117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:03.531307 26117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:03.534592 26117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:03.534794 26117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:03.535401 26117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:03.536278 26117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:03.536347 26117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.536419 26117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:03.536468 26117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.546087 26117 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:43413
I20260928 23:44:03.546095 26244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:43413 every 8 connection(s)
I20260928 23:44:03.546728 26117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:03.548267 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26117
I20260928 23:44:03.548400 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:03.548738 21953 external_mini_cluster.cc:1428] Setting key ad5d44c2c6b50d9224a233ed387a8d40
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:44:03.562309 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:03.548621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:60679 (local address 127.21.112.126:35331)
0928 23:44:03.548756 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:03.548760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:03.550361 (+ 1601us) server_negotiation.cc:407] Connection header received
0928 23:44:03.551295 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:03.551301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:03.551367 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:03.551466 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:03.552857 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.553431 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.554667 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.554905 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.555014 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:03.557807 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:03.557810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:03.557816 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:03.557848 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:03.559446 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:03.560190 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:03.560193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:03.560194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:03.560288 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:03.560719 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:03.560722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:03.560723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:03.561013 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:03.561163 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:03.561870 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 23:44:03.562037 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:44:03.563086 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:03.549132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:35331 (local address 127.21.112.65:60679)
0928 23:44:03.550179 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:03.550240 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:03.551081 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:03.551689 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:03.551697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:03.552163 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:03.552683 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:03.552698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.553597 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.553601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:03.554523 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:03.554532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.555055 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.555060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:03.555231 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:03.555871 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:03.555888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:03.557634 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:03.559576 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:03.559582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:03.559596 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:03.560023 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:03.560408 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:03.560410 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:03.560412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:03.560580 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:03.561218 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:03.561224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:03.561519 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 23:44:03.561763 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 23:44:03.562068 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":145,"threads_started":1}
I20260928 23:44:03.564597 26245 heartbeater.cc:347] Connected to a master server at 127.21.112.126:35331
I20260928 23:44:03.564851 26245 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:03.565344 26245 heartbeater.cc:511] Master 127.21.112.126:35331 requested a full tablet report, sending...
I20260928 23:44:03.567178 26055 ts_manager.cc:194] Registered new tserver with Master: 226633ba66a04650a369247d072292a9 (127.21.112.65:43413)
I20260928 23:44:03.568409 26055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:60679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:44:03.581264 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:35331
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:38583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:03.697419 26250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:03.697769 26250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:03.697821 26250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:03.701876 26250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:03.701941 26250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:03.702001 26250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:03.707564 26250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:03.708604 26250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:03.709764 26250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:03.720026 26255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:03.720062 26256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:03.720127 26258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:03.720412 26250 server_base.cc:1034] running on GCE node
I20260928 23:44:03.720984 26250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:03.721767 26250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:03.723078 26250 hybrid_clock.cc:648] HybridClock initialized: now 1790639043723052 us; error 40 us; skew 500 ppm
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:03.726187 26250 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:44:03.726975 26250 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:03.727041 26250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:03.727255 26250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:03.729427 26250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "5a23bcb6143342f0be4bcc5c2848ba28"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "3985508e6ede4c8e349a9e8614b63843"
server_key_iv: "19c0bbf4f26df4d5ff93d338c659fd42"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.730031 26250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "5a23bcb6143342f0be4bcc5c2848ba28"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "3985508e6ede4c8e349a9e8614b63843"
server_key_iv: "19c0bbf4f26df4d5ff93d338c659fd42"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.734215 26250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:44:03.736861 26264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.738441 26250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:03.738631 26250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "5a23bcb6143342f0be4bcc5c2848ba28"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "3985508e6ede4c8e349a9e8614b63843"
server_key_iv: "19c0bbf4f26df4d5ff93d338c659fd42"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.738770 26250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:03.756597 26250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:03.760268 26250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:03.760430 26250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:03.760980 26250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:03.761883 26250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:03.761940 26250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.761981 26250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:03.761996 26250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.771277 26250 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:39329
I20260928 23:44:03.771301 26377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:39329 every 8 connection(s)
I20260928 23:44:03.771900 26250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:03.778417 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26250
I20260928 23:44:03.778550 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:03.778861 21953 external_mini_cluster.cc:1428] Setting key 13af7aa444f466a41eb0b4ac3e9c1269
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:03.787828 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:03.773928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:42089 (local address 127.21.112.126:35331)
0928 23:44:03.774101 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:03.774106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:03.775461 (+ 1355us) server_negotiation.cc:407] Connection header received
0928 23:44:03.776421 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:03.776424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:03.776475 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:03.776567 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:03.777977 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.778899 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.780393 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.780767 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.780898 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:03.783906 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:03.783909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:03.783911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:03.783946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:03.785515 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:03.786143 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:03.786146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:03.786149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:03.786234 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:03.786631 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:03.786635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:03.786637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:03.786839 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:03.786962 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:03.787495 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 23:44:03.787629 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:44:03.788795 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:03.774356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:35331 (local address 127.21.112.66:42089)
0928 23:44:03.775261 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:03.775317 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:03.776186 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:03.776765 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:03.776772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:03.777224 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:03.777826 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:03.777841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.779042 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.779045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:03.780278 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:03.780290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:03.780937 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:03.780940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:03.781160 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:03.781845 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:03.781863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:03.783710 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:03.785697 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:03.785705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:03.785723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:03.786021 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:03.786331 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:03.786336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:03.786338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:03.786514 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:03.786952 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:03.786959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:03.787240 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 23:44:03.787458 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 23:44:03.787826 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":113,"threads_started":1}
I20260928 23:44:03.790096 26378 heartbeater.cc:347] Connected to a master server at 127.21.112.126:35331
I20260928 23:44:03.790354 26378 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:03.790833 26378 heartbeater.cc:511] Master 127.21.112.126:35331 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:44:03.792063 26055 ts_manager.cc:194] Registered new tserver with Master: 5a23bcb6143342f0be4bcc5c2848ba28 (127.21.112.66:39329)
I20260928 23:44:03.792732 26055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:42089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:44:03.807974 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:35331
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:38583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:03.926468 26383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:03.926743 26383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:03.926787 26383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:03.931183 26383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:03.931263 26383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:03.931329 26383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:03.937078 26383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:03.938267 26383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:03.939379 26383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:03.949347 26391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:03.949393 26383 server_base.cc:1034] running on GCE node
W20260928 23:44:03.949424 26389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:03.949411 26388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:03.950264 26383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:03.951036 26383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:03.952267 26383 hybrid_clock.cc:648] HybridClock initialized: now 1790639043952220 us; error 58 us; skew 500 ppm
Sep 28 23:44:03 dist-test-slave-c3s8 krb5kdc[26011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:03.955492 26383 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:44:03.956487 26383 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:03.956593 26383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:03.956874 26383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:03.959300 26383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "a87feda3d587444db018d55afa91d81e"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "188328629284332c54ac224e39912d0e"
server_key_iv: "c349f2d68f5bc5271fb06debf907afa9"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.959893 26383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "a87feda3d587444db018d55afa91d81e"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "188328629284332c54ac224e39912d0e"
server_key_iv: "c349f2d68f5bc5271fb06debf907afa9"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.963864 26383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:03.966533 26397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.968027 26383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:03.968178 26383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "a87feda3d587444db018d55afa91d81e"
format_stamp: "Formatted at 2026-09-28 23:44:03 on dist-test-slave-c3s8"
server_key: "188328629284332c54ac224e39912d0e"
server_key_iv: "c349f2d68f5bc5271fb06debf907afa9"
server_key_version: "encryptionkey@0"
I20260928 23:44:03.968310 26383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:03.987152 26383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:03.990643 26383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:03.990859 26383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:03.991472 26383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:03.992349 26383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:03.992419 26383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:03.992487 26383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:03.992533 26383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:04.002136 26383 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:37411
I20260928 23:44:04.002154 26510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:37411 every 8 connection(s)
I20260928 23:44:04.002807 26383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:04.005041 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26383
I20260928 23:44:04.005182 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:04.005486 21953 external_mini_cluster.cc:1428] Setting key 32a90248b8ae19067e86086413bb0724
Sep 28 23:44:04 dist-test-slave-c3s8 krb5kdc[26011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639043, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:04.018327 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.004841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:56507 (local address 127.21.112.126:35331)
0928 23:44:04.005007 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.005012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.006564 (+ 1552us) server_negotiation.cc:407] Connection header received
0928 23:44:04.007558 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.007561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.007615 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.007690 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:04.008880 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.009417 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.010693 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.010919 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.011026 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.013621 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:04.013624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:04.013627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:04.013660 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:04.015638 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:04.016381 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:04.016385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:04.016387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:04.016515 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:04.016976 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:04.016979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:04.016981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:04.017242 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:04.017373 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.018018 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.018160 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:44:04.019476 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.005298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:35331 (local address 127.21.112.67:56507)
0928 23:44:04.006418 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.006485 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.007356 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.007862 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.007869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.008245 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:04.008743 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.008757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.009562 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.009565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.010556 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.010566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.011072 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.011076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.011253 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.011871 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:04.011896 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:04.013378 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:04.015821 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:04.015828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:04.015863 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:04.016243 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:04.016637 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:04.016642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:04.016645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:04.016874 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:04.017371 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:04.017378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:04.017736 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.017985 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.018373 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":159,"threads_started":1}
I20260928 23:44:04.020694 26511 heartbeater.cc:347] Connected to a master server at 127.21.112.126:35331
I20260928 23:44:04.020931 26511 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:04.021376 26511 heartbeater.cc:511] Master 127.21.112.126:35331 requested a full tablet report, sending...
I20260928 23:44:04.022497 26055 ts_manager.cc:194] Registered new tserver with Master: a87feda3d587444db018d55afa91d81e (127.21.112.67:37411)
I20260928 23:44:04.023102 26055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:56507
I20260928 23:44:04.031860 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:44:04.042443 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.034071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.21.112.126:35331)
0928 23:44:04.034219 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.034224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.034407 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:44:04.034569 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.034571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.034619 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.034694 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:04.035546 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.036087 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.036961 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.037188 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.037272 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.038796 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:04.038799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:04.038801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:04.038827 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:04.040354 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:04.040815 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:04.040818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:04.040819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:04.040909 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:04.041283 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:04.041286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:04.041287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:04.041632 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:04.041752 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.042051 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.042244 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:44:04.046463 26055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:04.049739 26055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:04.066381 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.060540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:43413 (local address 127.0.0.1:37244)
0928 23:44:04.060928 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.060988 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.061139 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.061948 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.061951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.061978 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.062180 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.062206 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.063621 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.063624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.064700 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.064710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.065740 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.065745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.065866 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.065913 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.066041 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.066189 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":171,"spinlock_wait_cycles":71168,"thread_start_us":93,"threads_started":1}
I20260928 23:44:04.066402 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.060847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.21.112.65:43413)
0928 23:44:04.061437 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.061442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.061460 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:44:04.061522 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.061526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.061788 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.061950 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.062330 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.063476 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.064840 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.065495 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.065754 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.065946 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.066116 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.066230 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":75,"threads_started":1}
I20260928 23:44:04.067442 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.059756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37411 (local address 127.0.0.1:60194)
0928 23:44:04.060929 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.060986 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.061156 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.061948 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.061951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.061978 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.062128 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.062135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.063500 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.063504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.065328 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.065338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.066986 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.066990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.067089 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.067115 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.067203 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.067282 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":182,"spinlock_wait_cycles":2688,"thread_start_us":99,"threads_started":1}
I20260928 23:44:04.067656 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.060124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.21.112.67:37411)
0928 23:44:04.060574 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.060589 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.061074 (+ 485us) server_negotiation.cc:407] Connection header received
0928 23:44:04.061325 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.061332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.061612 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.061741 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.062259 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.063337 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.065489 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.066612 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.066746 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.066915 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.067287 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.067382 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":140,"threads_started":1}
I20260928 23:44:04.068789 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.059746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:39329 (local address 127.0.0.1:47018)
0928 23:44:04.060938 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.060988 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.061139 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.061772 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.061778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.061806 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.061992 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.062002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.063555 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.063561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.065325 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.065357 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.068364 (+ 3007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.068368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.068474 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.068497 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.068579 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.068655 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":205,"spinlock_wait_cycles":4480,"thread_start_us":129,"threads_started":1}
I20260928 23:44:04.069006 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.060485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.21.112.66:39329)
0928 23:44:04.061259 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.061272 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.061293 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:44:04.061371 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.061375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.061596 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.061748 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.062108 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.063388 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.065694 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.068228 (+ 2534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.068334 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.068444 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.068649 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.068747 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":107,"threads_started":1}
I20260928 23:44:04.070185 26179 tablet_service.cc:1467] Processing CreateTablet for tablet ed0047a7f3c645ed84208efed7f7f964 (DEFAULT_TABLE table=test-table [id=7656018036e34c449ce2704758e18021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:04.071430 26179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0047a7f3c645ed84208efed7f7f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:04.072831 26312 tablet_service.cc:1467] Processing CreateTablet for tablet ed0047a7f3c645ed84208efed7f7f964 (DEFAULT_TABLE table=test-table [id=7656018036e34c449ce2704758e18021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:04.072829 26445 tablet_service.cc:1467] Processing CreateTablet for tablet ed0047a7f3c645ed84208efed7f7f964 (DEFAULT_TABLE table=test-table [id=7656018036e34c449ce2704758e18021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:04.073984 26445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0047a7f3c645ed84208efed7f7f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:04.073977 26312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0047a7f3c645ed84208efed7f7f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:04.077970 26528 tablet_bootstrap.cc:492] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: Bootstrap starting.
I20260928 23:44:04.080438 26528 tablet_bootstrap.cc:654] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:04.081360 26528 log.cc:824] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:04.082208 26529 tablet_bootstrap.cc:492] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Bootstrap starting.
I20260928 23:44:04.084175 26529 tablet_bootstrap.cc:654] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:04.084739 26530 tablet_bootstrap.cc:492] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Bootstrap starting.
I20260928 23:44:04.085198 26529 log.cc:824] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:04.087126 26530 tablet_bootstrap.cc:654] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:04.087507 26528 tablet_bootstrap.cc:492] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: No bootstrap required, opened a new log
I20260928 23:44:04.087527 26529 tablet_bootstrap.cc:492] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: No bootstrap required, opened a new log
I20260928 23:44:04.087724 26528 ts_tablet_manager.cc:1397] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 23:44:04.087724 26529 ts_tablet_manager.cc:1397] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260928 23:44:04.088474 26530 log.cc:824] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:04.090652 26530 tablet_bootstrap.cc:492] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: No bootstrap required, opened a new log
I20260928 23:44:04.090890 26530 ts_tablet_manager.cc:1397] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 23:44:04.092041 26529 raft_consensus.cc:348] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.092355 26529 raft_consensus.cc:374] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:04.092411 26529 raft_consensus.cc:729] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a23bcb6143342f0be4bcc5c2848ba28, State: Initialized, Role: FOLLOWER
I20260928 23:44:04.092320 26528 raft_consensus.cc:348] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.092578 26528 raft_consensus.cc:374] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:04.092633 26528 raft_consensus.cc:729] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226633ba66a04650a369247d072292a9, State: Initialized, Role: FOLLOWER
I20260928 23:44:04.092967 26529 consensus_queue.cc:260] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.093012 26528 consensus_queue.cc:260] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.094043 26245 heartbeater.cc:503] Master 127.21.112.126:35331 was elected leader, sending a full tablet report...
I20260928 23:44:04.094146 26529 ts_tablet_manager.cc:1428] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 23:44:04.094146 26528 ts_tablet_manager.cc:1428] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 23:44:04.094519 26378 heartbeater.cc:503] Master 127.21.112.126:35331 was elected leader, sending a full tablet report...
I20260928 23:44:04.094853 26530 raft_consensus.cc:348] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.095170 26530 raft_consensus.cc:374] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:04.095244 26530 raft_consensus.cc:729] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87feda3d587444db018d55afa91d81e, State: Initialized, Role: FOLLOWER
I20260928 23:44:04.095976 26530 consensus_queue.cc:260] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.096705 26511 heartbeater.cc:503] Master 127.21.112.126:35331 was elected leader, sending a full tablet report...
I20260928 23:44:04.097554 26530 ts_tablet_manager.cc:1428] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260928 23:44:04.253757 26512 tablet.cc:2369] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:44:04.273101 26379 tablet.cc:2369] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:44:04.298048 26246 tablet.cc:2369] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:44:04.328195 26534 raft_consensus.cc:482] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:04.328486 26534 raft_consensus.cc:504] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.329766 26534 leader_election.cc:288] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a23bcb6143342f0be4bcc5c2848ba28 (127.21.112.66:39329), a87feda3d587444db018d55afa91d81e (127.21.112.67:37411)
I20260928 23:44:04.331004 26535 raft_consensus.cc:482] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:04.331176 26535 raft_consensus.cc:504] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.332472 26535 leader_election.cc:288] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a87feda3d587444db018d55afa91d81e (127.21.112.67:37411), 226633ba66a04650a369247d072292a9 (127.21.112.65:43413)
I20260928 23:44:04.334939 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.330179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:36549 (local address 127.21.112.66:39329)
0928 23:44:04.330340 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.330346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.330624 (+ 278us) server_negotiation.cc:407] Connection header received
0928 23:44:04.330758 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.330762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.330824 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.330914 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.331225 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.331891 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.333656 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.334284 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.334364 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.334414 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.334743 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.334817 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 23:44:04.334997 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.330296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:51373 (local address 127.21.112.67:37411)
0928 23:44:04.330410 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.330414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.330736 (+ 322us) server_negotiation.cc:407] Connection header received
0928 23:44:04.330852 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.330855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.330905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.330999 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.331309 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.332043 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.333676 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.334323 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.334405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.334452 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.334799 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.334868 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:44:04.335404 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.330085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37411 (local address 127.21.112.65:51373)
0928 23:44:04.330614 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.330630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.330745 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.331044 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.331049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.331065 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.331187 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.331197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.332210 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.332213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.333494 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.333509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.334546 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.334549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.334671 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.334691 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.335177 (+ 486us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.335254 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":25,"thread_start_us":106,"threads_started":1}
I20260928 23:44:04.336023 26465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0047a7f3c645ed84208efed7f7f964" candidate_uuid: "226633ba66a04650a369247d072292a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87feda3d587444db018d55afa91d81e" is_pre_election: true
I20260928 23:44:04.336331 26465 raft_consensus.cc:2454] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226633ba66a04650a369247d072292a9 in term 0.
I20260928 23:44:04.335737 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.330085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:39329 (local address 127.21.112.65:36549)
0928 23:44:04.330514 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.330527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.330639 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.330935 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.330938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.330959 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.331088 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.331095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.331997 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.332002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.333501 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.333514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.334506 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.334509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.334609 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.334631 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.335373 (+ 742us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.335445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":98,"thread_start_us":133,"threads_started":1}
I20260928 23:44:04.336872 26132 leader_election.cc:302] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226633ba66a04650a369247d072292a9, a87feda3d587444db018d55afa91d81e; no voters:
I20260928 23:44:04.337203 26534 raft_consensus.cc:2781] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:04.337309 26534 raft_consensus.cc:482] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:04.337356 26534 raft_consensus.cc:3037] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:04.337507 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.332653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37411 (local address 127.21.112.66:54183)
0928 23:44:04.332914 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.332929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.333012 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.333912 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.333915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.333933 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.334037 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.334042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.335119 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.335123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.336206 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.336216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.337064 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.337067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.337183 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.337203 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.337276 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.337343 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260928 23:44:04.337513 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.333295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:54183 (local address 127.21.112.67:37411)
0928 23:44:04.333703 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.333712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.333723 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:44:04.333755 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.333758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.333797 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.334173 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.334265 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.334994 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.336348 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.336960 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.337075 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.337123 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.337307 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.337386 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":132,"threads_started":1}
I20260928 23:44:04.337713 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.333241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:43571 (local address 127.21.112.65:43413)
0928 23:44:04.333336 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.333341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.333607 (+ 266us) server_negotiation.cc:407] Connection header received
0928 23:44:04.333778 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.333781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.333821 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.333863 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:04.334262 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.334985 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.336457 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.337077 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.337181 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.337223 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.337496 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.337606 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:44:04.338095 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.332998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:43413 (local address 127.21.112.66:43571)
0928 23:44:04.333500 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.333514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.333671 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.333986 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.333988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.334005 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:04.334139 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.334145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.335138 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.335143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.336309 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:04.336320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.337280 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.337284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:04.337370 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.337388 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.337943 (+ 555us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.337997 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260928 23:44:04.338480 26534 raft_consensus.cc:504] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.338505 26465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0047a7f3c645ed84208efed7f7f964" candidate_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87feda3d587444db018d55afa91d81e" is_pre_election: true
I20260928 23:44:04.338620 26465 raft_consensus.cc:2454] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a23bcb6143342f0be4bcc5c2848ba28 in term 0.
I20260928 23:44:04.338631 26199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0047a7f3c645ed84208efed7f7f964" candidate_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226633ba66a04650a369247d072292a9" is_pre_election: true
I20260928 23:44:04.338675 26332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0047a7f3c645ed84208efed7f7f964" candidate_uuid: "226633ba66a04650a369247d072292a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" is_pre_election: true
I20260928 23:44:04.338827 26199 raft_consensus.cc:2379] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a23bcb6143342f0be4bcc5c2848ba28 in current term 1: Already voted for candidate 226633ba66a04650a369247d072292a9 in this term.
I20260928 23:44:04.338863 26534 leader_election.cc:288] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5a23bcb6143342f0be4bcc5c2848ba28 (127.21.112.66:39329), a87feda3d587444db018d55afa91d81e (127.21.112.67:37411)
I20260928 23:44:04.338857 26332 raft_consensus.cc:2454] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226633ba66a04650a369247d072292a9 in term 0.
I20260928 23:44:04.339315 26331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0047a7f3c645ed84208efed7f7f964" candidate_uuid: "226633ba66a04650a369247d072292a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a23bcb6143342f0be4bcc5c2848ba28"
I20260928 23:44:04.339434 26465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0047a7f3c645ed84208efed7f7f964" candidate_uuid: "226633ba66a04650a369247d072292a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87feda3d587444db018d55afa91d81e"
I20260928 23:44:04.339433 26331 raft_consensus.cc:3037] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:04.339529 26465 raft_consensus.cc:3037] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:04.340039 26265 leader_election.cc:302] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a23bcb6143342f0be4bcc5c2848ba28, a87feda3d587444db018d55afa91d81e; no voters:
I20260928 23:44:04.340718 26465 raft_consensus.cc:2454] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226633ba66a04650a369247d072292a9 in term 1.
I20260928 23:44:04.340718 26331 raft_consensus.cc:2454] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226633ba66a04650a369247d072292a9 in term 1.
I20260928 23:44:04.340898 26535 raft_consensus.cc:2741] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 23:44:04.341073 26133 leader_election.cc:302] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226633ba66a04650a369247d072292a9, 5a23bcb6143342f0be4bcc5c2848ba28; no voters:
I20260928 23:44:04.341300 26534 raft_consensus.cc:2781] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:04.341503 26534 raft_consensus.cc:686] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [term 1 LEADER]: Becoming Leader. State: Replica: 226633ba66a04650a369247d072292a9, State: Running, Role: LEADER
I20260928 23:44:04.341863 26534 consensus_queue.cc:237] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } }
I20260928 23:44:04.345206 26054 catalog_manager.cc:5495] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 226633ba66a04650a369247d072292a9 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "226633ba66a04650a369247d072292a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226633ba66a04650a369247d072292a9" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43413 } health_report { overall_health: HEALTHY } } }
I20260928 23:44:04.360466 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.356683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.21.112.65:43413)
0928 23:44:04.356845 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.356850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.356866 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:44:04.357046 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.357049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.357144 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.357265 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:04.357561 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.358136 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.359028 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:04.359252 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:04.359360 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:04.359666 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:04.360098 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:04.360230 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.360270 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.360321 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:04.366710 26465 raft_consensus.cc:1261] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 1 FOLLOWER]: Refusing update from remote peer 226633ba66a04650a369247d072292a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:04.366734 26331 raft_consensus.cc:1261] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 1 FOLLOWER]: Refusing update from remote peer 226633ba66a04650a369247d072292a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:04.367411 26534 consensus_queue.cc:1035] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a23bcb6143342f0be4bcc5c2848ba28" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:04.367632 26542 consensus_queue.cc:1035] T ed0047a7f3c645ed84208efed7f7f964 P 226633ba66a04650a369247d072292a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a87feda3d587444db018d55afa91d81e" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:04.376210 26546 mvcc.cc:205] Tried to move back new op lower bound from 7334457525718495232 to 7334457525626757120. Current Snapshot: MvccSnapshot[applied={T|T < 7334457525718495232}]
I20260928 23:44:04.378639 26547 mvcc.cc:205] Tried to move back new op lower bound from 7334457525718495232 to 7334457525626757120. Current Snapshot: MvccSnapshot[applied={T|T < 7334457525718495232 or (T in {7334457525718495232})}]
I20260928 23:44:04.384115 26053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:04.384311 26053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:04.387123 26053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 298f3c0dd3f5445d9a9e9a44282d9248: Sending DeleteTablet for 3 replicas of tablet ed0047a7f3c645ed84208efed7f7f964
I20260928 23:44:04.387964 26312 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0047a7f3c645ed84208efed7f7f964 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:04 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:47018
I20260928 23:44:04.388049 26445 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0047a7f3c645ed84208efed7f7f964 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:04 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:60194
I20260928 23:44:04.388172 26179 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0047a7f3c645ed84208efed7f7f964 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:04 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37244
I20260928 23:44:04.388715 26556 tablet_replica.cc:330] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: stopping tablet replica
I20260928 23:44:04.388717 26555 tablet_replica.cc:330] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: stopping tablet replica
I20260928 23:44:04.389055 26556 raft_consensus.cc:2229] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:04.389070 26555 raft_consensus.cc:2229] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:04.389106 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26117
I20260928 23:44:04.389402 26555 raft_consensus.cc:2258] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:04.389403 26556 raft_consensus.cc:2258] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:04.391589 26555 ts_tablet_manager.cc:1905] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:04.392575 26556 ts_tablet_manager.cc:1905] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:04.395049 26556 ts_tablet_manager.cc:1918] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:04.395159 26556 log.cc:1196] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/ed0047a7f3c645ed84208efed7f7f964
I20260928 23:44:04.395447 26556 ts_tablet_manager.cc:1939] T ed0047a7f3c645ed84208efed7f7f964 P a87feda3d587444db018d55afa91d81e: Deleting consensus metadata
I20260928 23:44:04.395596 26555 ts_tablet_manager.cc:1918] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:04.395702 26555 log.cc:1196] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/ed0047a7f3c645ed84208efed7f7f964
I20260928 23:44:04.396027 26555 ts_tablet_manager.cc:1939] T ed0047a7f3c645ed84208efed7f7f964 P 5a23bcb6143342f0be4bcc5c2848ba28: Deleting consensus metadata
I20260928 23:44:04.396739 26039 catalog_manager.cc:4841] TS a87feda3d587444db018d55afa91d81e (127.21.112.67:37411): tablet ed0047a7f3c645ed84208efed7f7f964 (table test-table [id=7656018036e34c449ce2704758e18021]) successfully deleted
I20260928 23:44:04.397162 26040 catalog_manager.cc:4841] TS 5a23bcb6143342f0be4bcc5c2848ba28 (127.21.112.66:39329): tablet ed0047a7f3c645ed84208efed7f7f964 (table test-table [id=7656018036e34c449ce2704758e18021]) successfully deleted
W20260928 23:44:04.397408 26039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:43413 (error 108)
I20260928 23:44:04.397745 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26250
I20260928 23:44:04.398941 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.398571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:43413 (local address 127.0.0.1:37260)
0928 23:44:04.398740 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.398829 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:43413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 23:44:04.399138 26039 catalog_manager.cc:4584] TS 226633ba66a04650a369247d072292a9 (127.21.112.65:43413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed0047a7f3c645ed84208efed7f7f964: Network error: Client connection negotiation failed: client connection to 127.21.112.65:43413: connect: Connection refused (error 111)
I20260928 23:44:04.403692 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26383
I20260928 23:44:04.409446 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26024
2026-09-28T23:44:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2593 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T23:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:04Z Disabled control of system clock
I20260928 23:44:04.441236 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:42105
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:42105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:44:04.556108 26562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:04.556480 26562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:04.556550 26562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:04.560906 26562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:04.560988 26562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:04.561005 26562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:04.561022 26562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:04.561038 26562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:04.565987 26562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:42105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:42105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:04.567281 26562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:04.568766 26562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:04.577811 26567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:04.577968 26562 server_base.cc:1034] running on GCE node
W20260928 23:44:04.577819 26568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:04.577819 26570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:04.578758 26562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:04.579946 26562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:04.581174 26562 hybrid_clock.cc:648] HybridClock initialized: now 1790639044581157 us; error 30 us; skew 500 ppm
I20260928 23:44:04.583250 26562 webserver.cc:460] Webserver started at http://127.21.112.126:39825/ using document root <none> and password file <none>
I20260928 23:44:04.583858 26562 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:04.583950 26562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:04.584192 26562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:04.586443 26562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "4253248241f847d393d7c9d0d31856e8"
format_stamp: "Formatted at 2026-09-28 23:44:04 on dist-test-slave-c3s8"
server_key: "4f7983b64e206fea3a1955e78da01db8"
server_key_iv: "9e4515148149baf7cfca7bfd795f5e5a"
server_key_version: "encryptionkey@0"
I20260928 23:44:04.587056 26562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "4253248241f847d393d7c9d0d31856e8"
format_stamp: "Formatted at 2026-09-28 23:44:04 on dist-test-slave-c3s8"
server_key: "4f7983b64e206fea3a1955e78da01db8"
server_key_iv: "9e4515148149baf7cfca7bfd795f5e5a"
server_key_version: "encryptionkey@0"
I20260928 23:44:04.590994 26562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:04.594554 26576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:04.595960 26562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:44:04.596123 26562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "4253248241f847d393d7c9d0d31856e8"
format_stamp: "Formatted at 2026-09-28 23:44:04 on dist-test-slave-c3s8"
server_key: "4f7983b64e206fea3a1955e78da01db8"
server_key_iv: "9e4515148149baf7cfca7bfd795f5e5a"
server_key_version: "encryptionkey@0"
I20260928 23:44:04.596256 26562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:04.601621 26562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:04.602310 26562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:04.602497 26562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:04.609700 26562 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:42105
I20260928 23:44:04.609710 26628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:42105 every 8 connection(s)
I20260928 23:44:04.610811 26562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:04.614256 26629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:04.617645 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26562
I20260928 23:44:04.617779 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:04.618083 21953 external_mini_cluster.cc:1428] Setting key 6553a99c640a45c010337fcda78a3792
I20260928 23:44:04.620728 26629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8: Bootstrap starting.
I20260928 23:44:04.623247 26629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:04.624135 26629 log.cc:824] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:04.624567 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.619689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.21.112.126:42105)
0928 23:44:04.620504 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.620520 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.620557 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:44:04.621415 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.621436 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.621911 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.622304 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:04.623109 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:04.623113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:04.623125 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:04.623135 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:04.623189 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:04.623212 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:04.623332 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.623478 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.623764 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":173,"threads_started":1}
I20260928 23:44:04.626225 26629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8: No bootstrap required, opened a new log
I20260928 23:44:04.630498 26629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253248241f847d393d7c9d0d31856e8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42105 } }
I20260928 23:44:04.630745 26629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:04.630801 26629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4253248241f847d393d7c9d0d31856e8, State: Initialized, Role: FOLLOWER
I20260928 23:44:04.631274 26629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253248241f847d393d7c9d0d31856e8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42105 } }
I20260928 23:44:04.631407 26629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:04.631484 26629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:04.631599 26629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:04.632602 26629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253248241f847d393d7c9d0d31856e8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42105 } }
I20260928 23:44:04.632956 26629 leader_election.cc:302] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4253248241f847d393d7c9d0d31856e8; no voters:
I20260928 23:44:04.633296 26629 leader_election.cc:288] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:04.633462 26634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:04.633949 26634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [term 1 LEADER]: Becoming Leader. State: Replica: 4253248241f847d393d7c9d0d31856e8, State: Running, Role: LEADER
I20260928 23:44:04.634297 26634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253248241f847d393d7c9d0d31856e8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42105 } }
I20260928 23:44:04.634648 26629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:04.635471 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4253248241f847d393d7c9d0d31856e8. Latest consensus state: current_term: 1 leader_uuid: "4253248241f847d393d7c9d0d31856e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253248241f847d393d7c9d0d31856e8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42105 } } }
I20260928 23:44:04.635604 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:04.635880 26635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4253248241f847d393d7c9d0d31856e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4253248241f847d393d7c9d0d31856e8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42105 } } }
I20260928 23:44:04.635974 26635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:04.636070 26643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:04.639336 26643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:04.644960 26643 catalog_manager.cc:1347] Generated new cluster ID: cf83be25d67548cfa35e9c9f53151153
I20260928 23:44:04.645054 26643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:04.659560 26643 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:04.659953 26643 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:04.668720 26643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8: Generated new TSK 0
I20260928 23:44:04.669687 26643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:44:04.682999 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:42105
--builtin_ntp_servers=127.21.112.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:44:04.799131 26653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:04.799413 26653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:04.799454 26653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:04.803778 26653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:04.803841 26653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:04.803931 26653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:04.808787 26653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:42105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:04.810011 26653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:04.811447 26653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:04.821617 26659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:04.821691 26653 server_base.cc:1034] running on GCE node
W20260928 23:44:04.821617 26661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:04.821631 26658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:04.822489 26653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:04.823153 26653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:04.824326 26653 hybrid_clock.cc:648] HybridClock initialized: now 1790639044824283 us; error 56 us; skew 500 ppm
I20260928 23:44:04.826315 26653 webserver.cc:460] Webserver started at http://127.21.112.65:41229/ using document root <none> and password file <none>
I20260928 23:44:04.826920 26653 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:04.826989 26653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:04.827207 26653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:04.829349 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "9303adb83318439c9353701b51c35e78"
format_stamp: "Formatted at 2026-09-28 23:44:04 on dist-test-slave-c3s8"
server_key: "b22cb2879086c6147c7ec96e2793cac4"
server_key_iv: "e239153e6b2a9dd0be92b9c2fd0259e6"
server_key_version: "encryptionkey@0"
I20260928 23:44:04.829972 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "9303adb83318439c9353701b51c35e78"
format_stamp: "Formatted at 2026-09-28 23:44:04 on dist-test-slave-c3s8"
server_key: "b22cb2879086c6147c7ec96e2793cac4"
server_key_iv: "e239153e6b2a9dd0be92b9c2fd0259e6"
server_key_version: "encryptionkey@0"
I20260928 23:44:04.833870 26653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:04.836663 26667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:04.838057 26653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:04.838186 26653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "9303adb83318439c9353701b51c35e78"
format_stamp: "Formatted at 2026-09-28 23:44:04 on dist-test-slave-c3s8"
server_key: "b22cb2879086c6147c7ec96e2793cac4"
server_key_iv: "e239153e6b2a9dd0be92b9c2fd0259e6"
server_key_version: "encryptionkey@0"
I20260928 23:44:04.838302 26653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:04.843041 26653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:04.843660 26653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:04.843838 26653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:04.844373 26653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:04.845289 26653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:04.845357 26653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:04.845422 26653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:04.845468 26653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:04.854568 26653 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:43133
I20260928 23:44:04.854619 26780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:43133 every 8 connection(s)
I20260928 23:44:04.855608 26653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:04.859953 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26653
I20260928 23:44:04.860051 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:04.860311 21953 external_mini_cluster.cc:1428] Setting key 980698adbaacec3e5654e3440db9e0ee
I20260928 23:44:04.862339 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:42105
--builtin_ntp_servers=127.21.112.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:44:04.862463 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.857710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:46411 (local address 127.21.112.126:42105)
0928 23:44:04.857855 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:04.857859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:04.859014 (+ 1155us) server_negotiation.cc:407] Connection header received
0928 23:44:04.860556 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:04.860559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:04.860604 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:04.860703 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:04.861931 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:04.861935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:04.861938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:04.861956 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:04.861977 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:04.861987 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:04.862102 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:04.862269 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:44:04.862320 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:44:04.863507 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:04.858122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:42105 (local address 127.21.112.65:46411)
0928 23:44:04.858850 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:04.858892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:04.860311 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:04.860874 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:04.860881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:04.860982 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:04.861602 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:04.861606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:04.861678 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:04.861778 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:04.862097 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:04.862108 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:04.862117 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:44:04.862317 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 23:44:04.862590 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":116,"threads_started":1}
I20260928 23:44:04.864848 26781 heartbeater.cc:347] Connected to a master server at 127.21.112.126:42105
I20260928 23:44:04.865127 26781 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:04.865648 26781 heartbeater.cc:511] Master 127.21.112.126:42105 requested a full tablet report, sending...
I20260928 23:44:04.867558 26593 ts_manager.cc:194] Registered new tserver with Master: 9303adb83318439c9353701b51c35e78 (127.21.112.65:43133)
W20260928 23:44:04.979282 26784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:04.979579 26784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:04.979621 26784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:04.983772 26784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:04.983837 26784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:04.983899 26784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:04.988750 26784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:42105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:04.989928 26784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:04.991353 26784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:05.001593 26792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:05.001643 26789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:05.001602 26790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:05.001914 26784 server_base.cc:1034] running on GCE node
I20260928 23:44:05.002455 26784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:05.003186 26784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:05.004422 26784 hybrid_clock.cc:648] HybridClock initialized: now 1790639045004395 us; error 33 us; skew 500 ppm
I20260928 23:44:05.006645 26784 webserver.cc:460] Webserver started at http://127.21.112.66:33753/ using document root <none> and password file <none>
I20260928 23:44:05.007380 26784 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:05.007494 26784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:05.007748 26784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:05.010281 26784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "f5ea1628bbbf4be998bf0b21faef80a7"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "c4a53e636a16ba883b202db0e047b59e"
server_key_iv: "95c5fbd5992362822a49e18dcc6ecda5"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.010942 26784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "f5ea1628bbbf4be998bf0b21faef80a7"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "c4a53e636a16ba883b202db0e047b59e"
server_key_iv: "95c5fbd5992362822a49e18dcc6ecda5"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.015223 26784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:05.018297 26798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.019774 26784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:05.019943 26784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "f5ea1628bbbf4be998bf0b21faef80a7"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "c4a53e636a16ba883b202db0e047b59e"
server_key_iv: "95c5fbd5992362822a49e18dcc6ecda5"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.020080 26784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:05.025244 26784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:05.026036 26784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:05.026242 26784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:05.026835 26784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:05.027846 26784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:05.027930 26784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.028019 26784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:05.028071 26784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.038491 26784 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:42025
I20260928 23:44:05.038513 26911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:42025 every 8 connection(s)
I20260928 23:44:05.039608 26784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:05.046358 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.041679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:44661 (local address 127.21.112.126:42105)
0928 23:44:05.041820 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.041824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.043123 (+ 1299us) server_negotiation.cc:407] Connection header received
0928 23:44:05.044540 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.044544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.044592 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.044697 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.045881 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.045885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.045887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.045905 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.045926 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.045934 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.046100 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.046168 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.046214 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:05.047245 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.042194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:42105 (local address 127.21.112.66:44661)
0928 23:44:05.042970 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.043014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.044311 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.044878 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.044885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.044979 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.045602 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.045604 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.045648 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.045720 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.046016 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.046025 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.046031 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.046207 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.046456 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":136,"threads_started":1}
I20260928 23:44:05.048419 26912 heartbeater.cc:347] Connected to a master server at 127.21.112.126:42105
I20260928 23:44:05.048635 26912 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:05.049108 26912 heartbeater.cc:511] Master 127.21.112.126:42105 requested a full tablet report, sending...
I20260928 23:44:05.049805 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26784
I20260928 23:44:05.050016 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:05.050278 26593 ts_manager.cc:194] Registered new tserver with Master: f5ea1628bbbf4be998bf0b21faef80a7 (127.21.112.66:42025)
I20260928 23:44:05.050359 21953 external_mini_cluster.cc:1428] Setting key ee8f1449403c90a2110a079aca6d9fb4
I20260928 23:44:05.052268 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:42105
--builtin_ntp_servers=127.21.112.84:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:44:05.168380 26915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:05.168690 26915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:05.168766 26915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:05.173700 26915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:05.173774 26915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:05.173842 26915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:05.179381 26915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:42105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:05.180578 26915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:05.182173 26915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:05.192658 26920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:05.192682 26921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:05.192662 26923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:05.192934 26915 server_base.cc:1034] running on GCE node
I20260928 23:44:05.193517 26915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:05.194257 26915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:05.195426 26915 hybrid_clock.cc:648] HybridClock initialized: now 1790639045195409 us; error 33 us; skew 500 ppm
I20260928 23:44:05.197441 26915 webserver.cc:460] Webserver started at http://127.21.112.67:45477/ using document root <none> and password file <none>
I20260928 23:44:05.198067 26915 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:05.198132 26915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:05.198344 26915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:05.200598 26915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "ee037ec129364ff9afa170f090e7ceb3"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "de251dc5522ece1fc0a30db193fdb462"
server_key_iv: "324dbb26e31053ac0cb1c93122676437"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.201210 26915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "ee037ec129364ff9afa170f090e7ceb3"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "de251dc5522ece1fc0a30db193fdb462"
server_key_iv: "324dbb26e31053ac0cb1c93122676437"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.205230 26915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 23:44:05.208364 26929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.209848 26915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:44:05.209978 26915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "ee037ec129364ff9afa170f090e7ceb3"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "de251dc5522ece1fc0a30db193fdb462"
server_key_iv: "324dbb26e31053ac0cb1c93122676437"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.210094 26915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:05.214998 26915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:05.215695 26915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:05.215883 26915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:05.216475 26915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:05.217429 26915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:05.217474 26915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.217536 26915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:05.217630 26915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.227300 26915 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:40997
I20260928 23:44:05.227322 27043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:40997 every 8 connection(s)
I20260928 23:44:05.228264 26915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:05.229103 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 26915
I20260928 23:44:05.229244 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:05.229672 21953 external_mini_cluster.cc:1428] Setting key f40f37ef7804e435ea89279bb9d79e48
I20260928 23:44:05.235232 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.230288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:53633 (local address 127.21.112.126:42105)
0928 23:44:05.230426 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.230431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.231859 (+ 1428us) server_negotiation.cc:407] Connection header received
0928 23:44:05.233368 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.233371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.233410 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.233493 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.234651 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.234654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.234656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.234668 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.234689 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.234697 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.234776 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.234990 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.235028 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:44:05.236291 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.230820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:42105 (local address 127.21.112.67:53633)
0928 23:44:05.231702 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.231760 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.233148 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.233696 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.233704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.233799 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.234356 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.234358 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.234398 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.234467 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.234841 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.234849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.234854 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.235063 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.235321 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":161,"threads_started":1}
I20260928 23:44:05.237525 27044 heartbeater.cc:347] Connected to a master server at 127.21.112.126:42105
I20260928 23:44:05.237756 27044 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:05.238161 27044 heartbeater.cc:511] Master 127.21.112.126:42105 requested a full tablet report, sending...
I20260928 23:44:05.239116 26593 ts_manager.cc:194] Registered new tserver with Master: ee037ec129364ff9afa170f090e7ceb3 (127.21.112.67:40997)
I20260928 23:44:05.243854 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:44:05.247491 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.245916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.21.112.126:42105)
0928 23:44:05.246093 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.246096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.246168 (+ 72us) server_negotiation.cc:407] Connection header received
0928 23:44:05.246280 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.246282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.246318 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.246382 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.247119 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.247122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.247124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.247135 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.247153 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.247160 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.247226 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.247300 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.247334 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:44:05.250517 26593 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:05.253970 26593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:05.267419 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.263439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42025 (local address 127.0.0.1:45254)
0928 23:44:05.264334 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.264393 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.264498 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.265107 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.265114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.265199 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.265913 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.265917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.265942 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.266134 (+ 192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.267035 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.267046 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.267052 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.267179 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.267252 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":91,"threads_started":1}
I20260928 23:44:05.267431 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.264211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.21.112.66:42025)
0928 23:44:05.264589 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.264607 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.264634 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:44:05.264690 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.264696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.264917 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.265084 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.266295 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.266304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.266321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.266338 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.266385 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.266931 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.267057 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.267197 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.267272 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":166,"threads_started":1}
I20260928 23:44:05.267786 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.263439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:43133 (local address 127.0.0.1:50324)
0928 23:44:05.265743 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.265758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.265850 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.266239 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.266241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.266258 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.266916 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.266918 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.266928 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.266954 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.267400 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.267402 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.267406 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.267472 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.267504 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2236,"mutex_wait_us":1475,"thread_start_us":116,"threads_started":1}
I20260928 23:44:05.267802 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.264637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.21.112.67:40997)
0928 23:44:05.265087 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.265100 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.265124 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:44:05.265172 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.265177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.265375 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.265555 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.266906 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.266913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.266925 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.266938 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.266980 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.267015 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.267123 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.267584 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.267661 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260928 23:44:05.267486 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.263766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:40997 (local address 127.0.0.1:37332)
0928 23:44:05.264334 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.264441 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.264497 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.266223 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.266226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.266243 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.266696 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.266699 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.266711 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.266744 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.267164 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.267167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.267170 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.267243 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.267298 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":128,"threads_started":1}
I20260928 23:44:05.268394 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.263693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.21.112.65:43133)
0928 23:44:05.264204 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.264220 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.265846 (+ 1626us) server_negotiation.cc:407] Connection header received
0928 23:44:05.265946 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.265952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.266110 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.266250 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.267158 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.267164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.267175 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.267188 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.267233 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.267269 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.267361 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.268179 (+ 818us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.268251 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":109,"threads_started":1}
I20260928 23:44:05.270841 26715 tablet_service.cc:1467] Processing CreateTablet for tablet 80fb59b2bd1c40ca94ee7a79be903adb (DEFAULT_TABLE table=test-table [id=576ee3ada902456a949217de83026c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:05.270915 26846 tablet_service.cc:1467] Processing CreateTablet for tablet 80fb59b2bd1c40ca94ee7a79be903adb (DEFAULT_TABLE table=test-table [id=576ee3ada902456a949217de83026c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:05.270913 26977 tablet_service.cc:1467] Processing CreateTablet for tablet 80fb59b2bd1c40ca94ee7a79be903adb (DEFAULT_TABLE table=test-table [id=576ee3ada902456a949217de83026c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:05.271970 26715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fb59b2bd1c40ca94ee7a79be903adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:05.271971 26977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fb59b2bd1c40ca94ee7a79be903adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:05.271971 26846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fb59b2bd1c40ca94ee7a79be903adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:05.278318 27061 tablet_bootstrap.cc:492] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: Bootstrap starting.
I20260928 23:44:05.279832 27062 tablet_bootstrap.cc:492] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: Bootstrap starting.
I20260928 23:44:05.279832 27063 tablet_bootstrap.cc:492] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Bootstrap starting.
I20260928 23:44:05.281345 27061 tablet_bootstrap.cc:654] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:05.282091 27062 tablet_bootstrap.cc:654] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:05.282091 27063 tablet_bootstrap.cc:654] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:05.282193 27061 log.cc:824] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:05.282956 27063 log.cc:824] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:05.282956 27062 log.cc:824] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:05.284368 27061 tablet_bootstrap.cc:492] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: No bootstrap required, opened a new log
I20260928 23:44:05.284585 27061 ts_tablet_manager.cc:1397] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 23:44:05.285008 27062 tablet_bootstrap.cc:492] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: No bootstrap required, opened a new log
I20260928 23:44:05.285200 27062 ts_tablet_manager.cc:1397] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:44:05.285331 27063 tablet_bootstrap.cc:492] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: No bootstrap required, opened a new log
I20260928 23:44:05.285504 27063 ts_tablet_manager.cc:1397] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:44:05.288700 27061 raft_consensus.cc:348] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.288964 27061 raft_consensus.cc:374] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:05.289036 27061 raft_consensus.cc:729] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9303adb83318439c9353701b51c35e78, State: Initialized, Role: FOLLOWER
I20260928 23:44:05.289505 27061 consensus_queue.cc:260] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.289939 27063 raft_consensus.cc:348] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.290201 27063 raft_consensus.cc:374] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:05.290268 27063 raft_consensus.cc:729] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee037ec129364ff9afa170f090e7ceb3, State: Initialized, Role: FOLLOWER
I20260928 23:44:05.290275 26781 heartbeater.cc:503] Master 127.21.112.126:42105 was elected leader, sending a full tablet report...
I20260928 23:44:05.290659 27061 ts_tablet_manager.cc:1428] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260928 23:44:05.289939 27062 raft_consensus.cc:348] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.290759 27063 consensus_queue.cc:260] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.290910 27062 raft_consensus.cc:374] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:05.290987 27062 raft_consensus.cc:729] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ea1628bbbf4be998bf0b21faef80a7, State: Initialized, Role: FOLLOWER
I20260928 23:44:05.291388 27062 consensus_queue.cc:260] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.291759 27063 ts_tablet_manager.cc:1428] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:44:05.291932 27044 heartbeater.cc:503] Master 127.21.112.126:42105 was elected leader, sending a full tablet report...
I20260928 23:44:05.292325 27062 ts_tablet_manager.cc:1428] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 23:44:05.292649 26912 heartbeater.cc:503] Master 127.21.112.126:42105 was elected leader, sending a full tablet report...
W20260928 23:44:05.356873 26782 tablet.cc:2369] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:44:05.427069 27069 raft_consensus.cc:482] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:05.427314 27069 raft_consensus.cc:504] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.428597 27069 leader_election.cc:288] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9303adb83318439c9353701b51c35e78 (127.21.112.65:43133), ee037ec129364ff9afa170f090e7ceb3 (127.21.112.67:40997)
I20260928 23:44:05.430720 27056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.428979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:60703 (local address 127.21.112.65:43133)
0928 23:44:05.429115 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.429119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.429424 (+ 305us) server_negotiation.cc:407] Connection header received
0928 23:44:05.429601 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.429604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.429638 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.429702 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.430331 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.430334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.430337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.430350 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.430371 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.430383 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.430471 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.430552 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.430589 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:44:05.430720 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.428993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:55875 (local address 127.21.112.67:40997)
0928 23:44:05.429137 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.429143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.429161 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:44:05.429329 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.429331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.429367 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.429463 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.430278 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.430283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.430286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.430303 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.430326 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.430336 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.430433 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.430545 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.430589 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:44:05.430722 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.428889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:40997 (local address 127.21.112.66:55875)
0928 23:44:05.429095 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.429111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.429210 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.429521 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.429527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.429567 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.430066 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.430067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.430078 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.430126 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.430440 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.430445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.430449 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.430532 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.430599 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 23:44:05.430745 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.428896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:43133 (local address 127.21.112.66:60703)
0928 23:44:05.429324 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.429337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.429467 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.429744 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.429750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.429768 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:05.430179 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:05.430183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:05.430194 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:05.430219 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:05.430460 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:05.430463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:05.430466 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:44:05.430548 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:44:05.430580 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":21,"thread_start_us":118,"threads_started":1}
I20260928 23:44:05.431430 26998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fb59b2bd1c40ca94ee7a79be903adb" candidate_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee037ec129364ff9afa170f090e7ceb3" is_pre_election: true
I20260928 23:44:05.431558 26735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fb59b2bd1c40ca94ee7a79be903adb" candidate_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9303adb83318439c9353701b51c35e78" is_pre_election: true
I20260928 23:44:05.431757 26998 raft_consensus.cc:2454] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ea1628bbbf4be998bf0b21faef80a7 in term 0.
I20260928 23:44:05.431860 26735 raft_consensus.cc:2454] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ea1628bbbf4be998bf0b21faef80a7 in term 0.
I20260928 23:44:05.432284 26802 leader_election.cc:302] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee037ec129364ff9afa170f090e7ceb3, f5ea1628bbbf4be998bf0b21faef80a7; no voters:
I20260928 23:44:05.432556 27069 raft_consensus.cc:2781] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:05.432642 27069 raft_consensus.cc:482] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:05.432695 27069 raft_consensus.cc:3037] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:05.433866 27069 raft_consensus.cc:504] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.434203 27069 leader_election.cc:288] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [CANDIDATE]: Term 1 election: Requested vote from peers 9303adb83318439c9353701b51c35e78 (127.21.112.65:43133), ee037ec129364ff9afa170f090e7ceb3 (127.21.112.67:40997)
I20260928 23:44:05.434589 26998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fb59b2bd1c40ca94ee7a79be903adb" candidate_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee037ec129364ff9afa170f090e7ceb3"
I20260928 23:44:05.434528 26735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fb59b2bd1c40ca94ee7a79be903adb" candidate_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9303adb83318439c9353701b51c35e78"
I20260928 23:44:05.434726 26998 raft_consensus.cc:3037] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:05.434738 26735 raft_consensus.cc:3037] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:05.435947 26735 raft_consensus.cc:2454] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ea1628bbbf4be998bf0b21faef80a7 in term 1.
I20260928 23:44:05.435956 26998 raft_consensus.cc:2454] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ea1628bbbf4be998bf0b21faef80a7 in term 1.
I20260928 23:44:05.436255 26802 leader_election.cc:302] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee037ec129364ff9afa170f090e7ceb3, f5ea1628bbbf4be998bf0b21faef80a7; no voters:
I20260928 23:44:05.436487 27069 raft_consensus.cc:2781] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:05.436707 27069 raft_consensus.cc:686] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 1 LEADER]: Becoming Leader. State: Replica: f5ea1628bbbf4be998bf0b21faef80a7, State: Running, Role: LEADER
I20260928 23:44:05.437049 27069 consensus_queue.cc:237] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } }
I20260928 23:44:05.440284 26593 catalog_manager.cc:5495] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ea1628bbbf4be998bf0b21faef80a7 (127.21.112.66). New cstate: current_term: 1 leader_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ea1628bbbf4be998bf0b21faef80a7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 } health_report { overall_health: UNKNOWN } } }
I20260928 23:44:05.461272 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.459581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.21.112.66:42025)
0928 23:44:05.459746 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.459751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.459765 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:05.459846 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.459848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.459911 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.459990 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.460680 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.460683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.460685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.460697 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.460717 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.460725 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.460856 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.461029 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.461077 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 23:44:05.467929 26998 raft_consensus.cc:1261] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 1 FOLLOWER]: Refusing update from remote peer f5ea1628bbbf4be998bf0b21faef80a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:05.467929 26735 raft_consensus.cc:1261] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 1 FOLLOWER]: Refusing update from remote peer f5ea1628bbbf4be998bf0b21faef80a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:05.468700 27071 consensus_queue.cc:1035] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee037ec129364ff9afa170f090e7ceb3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:05.468997 27069 consensus_queue.cc:1035] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9303adb83318439c9353701b51c35e78" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:05.477684 27074 mvcc.cc:205] Tried to move back new op lower bound from 7334457530229149696 to 7334457530113134592. Current Snapshot: MvccSnapshot[applied={T|T < 7334457530229149696}]
W20260928 23:44:05.479430 27045 tablet.cc:2369] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:44:05.487375 26591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:05.487598 26591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:05.490105 26591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4253248241f847d393d7c9d0d31856e8: Sending DeleteTablet for 3 replicas of tablet 80fb59b2bd1c40ca94ee7a79be903adb
I20260928 23:44:05.490931 26846 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fb59b2bd1c40ca94ee7a79be903adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:05 UTC) from {username='slave'} at 127.0.0.1:45254
I20260928 23:44:05.490931 26715 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fb59b2bd1c40ca94ee7a79be903adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:05 UTC) from {username='slave'} at 127.0.0.1:50324
I20260928 23:44:05.491117 26977 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fb59b2bd1c40ca94ee7a79be903adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:05 UTC) from {username='slave'} at 127.0.0.1:37332
I20260928 23:44:05.491515 27084 tablet_replica.cc:330] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78: stopping tablet replica
I20260928 23:44:05.491626 27085 tablet_replica.cc:330] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: stopping tablet replica
I20260928 23:44:05.491943 27084 raft_consensus.cc:2229] T 80fb59b2bd1c40ca94ee7a79be903adb P 9303adb83318439c9353701b51c35e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:05.491943 27085 raft_consensus.cc:2229] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:05.492056 27083 tablet_replica.cc:330] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: stopping tablet replica
I20260928 23:44:05.492283 27085 raft_consensus.cc:2258] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:05.492354 27083 raft_consensus.cc:2229] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:05.492401 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26653
I20260928 23:44:05.493098 27083 raft_consensus.cc:2258] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:05.496119 27085 ts_tablet_manager.cc:1905] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:05.498613 27083 ts_tablet_manager.cc:1905] T 80fb59b2bd1c40ca94ee7a79be903adb P f5ea1628bbbf4be998bf0b21faef80a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:05.498926 27085 ts_tablet_manager.cc:1918] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:05.499037 27085 log.cc:1196] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/80fb59b2bd1c40ca94ee7a79be903adb
I20260928 23:44:05.499423 27085 ts_tablet_manager.cc:1939] T 80fb59b2bd1c40ca94ee7a79be903adb P ee037ec129364ff9afa170f090e7ceb3: Deleting consensus metadata
W20260928 23:44:05.499922 26577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:43133 (error 108)
I20260928 23:44:05.500747 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26784
I20260928 23:44:05.502434 26580 catalog_manager.cc:4841] TS ee037ec129364ff9afa170f090e7ceb3 (127.21.112.67:40997): tablet 80fb59b2bd1c40ca94ee7a79be903adb (table test-table [id=576ee3ada902456a949217de83026c70]) successfully deleted
I20260928 23:44:05.502954 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.502577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:43133 (local address 127.0.0.1:50336)
0928 23:44:05.502740 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.502836 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 23:44:05.503162 26577 catalog_manager.cc:4584] TS 9303adb83318439c9353701b51c35e78 (127.21.112.65:43133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80fb59b2bd1c40ca94ee7a79be903adb: Network error: Client connection negotiation failed: client connection to 127.21.112.65:43133: connect: Connection refused (error 111)
I20260928 23:44:05.507539 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26915
I20260928 23:44:05.509205 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.507757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42025 (local address 127.0.0.1:45272)
0928 23:44:05.509010 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.509093 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.66:42025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1135}
I20260928 23:44:05.513306 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 26562
2026-09-28T23:44:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1102 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 23:44:05.535275 21953 mini_oidc.cc:152] Starting JWKS server
I20260928 23:44:05.536834 21953 webserver.cc:460] Webserver started at https://127.0.0.1:36867/ using document root <none> and password file <none>
I20260928 23:44:05.536988 21953 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:44:05.537216 21953 webserver.cc:460] Webserver started at http://127.0.0.1:42117/ using document root <none> and password file <none>
2026-09-28T23:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:05Z Disabled control of system clock
I20260928 23:44:05.546074 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:40301
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:40301
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36867/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/testchainca.cert with env {}
W20260928 23:44:05.662513 27092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:05.662802 27092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:05.662843 27092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:05.666980 27092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:05.667058 27092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:05.667074 27092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:05.667092 27092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:05.667109 27092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:44:05.667124 27092 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:44:05.667138 27092 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36867/jwks.json
I20260928 23:44:05.672685 27092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:40301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36867/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:05.673887 27092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:05.674914 27092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:05.683959 27100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:05.683972 27097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:05.683954 27098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:05.684301 27092 server_base.cc:1034] running on GCE node
I20260928 23:44:05.684954 27092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:05.686287 27092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:05.687486 27092 hybrid_clock.cc:648] HybridClock initialized: now 1790639045687467 us; error 37 us; skew 500 ppm
I20260928 23:44:05.689455 27092 webserver.cc:460] Webserver started at http://127.21.112.126:39227/ using document root <none> and password file <none>
I20260928 23:44:05.690106 27092 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:05.690197 27092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:05.690425 27092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:05.692566 27092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "f3b9837a871d4539bfe8196571484f13"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "aa34ff7cea6ccc83126cfb0f94012a0c"
server_key_iv: "38f1d5f585ded32fd464e116ee85a129"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.693190 27092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "f3b9837a871d4539bfe8196571484f13"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "aa34ff7cea6ccc83126cfb0f94012a0c"
server_key_iv: "38f1d5f585ded32fd464e116ee85a129"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.697191 27092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:44:05.699993 27106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:05.701529 27092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:05.701757 27092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "f3b9837a871d4539bfe8196571484f13"
format_stamp: "Formatted at 2026-09-28 23:44:05 on dist-test-slave-c3s8"
server_key: "aa34ff7cea6ccc83126cfb0f94012a0c"
server_key_iv: "38f1d5f585ded32fd464e116ee85a129"
server_key_version: "encryptionkey@0"
I20260928 23:44:05.701901 27092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:05.735455 27092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:05.736251 27092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:05.736481 27092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:05.744383 27092 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:40301
I20260928 23:44:05.744438 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:40301 every 8 connection(s)
I20260928 23:44:05.745468 27092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:05.749087 27159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:05.752781 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27092
I20260928 23:44:05.752939 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:05.753228 21953 external_mini_cluster.cc:1428] Setting key 801ed556c046e6a93846d125be2b0026
I20260928 23:44:05.755826 27159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13: Bootstrap starting.
I20260928 23:44:05.758322 27159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:05.759122 27159 log.cc:824] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:05.761298 27159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13: No bootstrap required, opened a new log
I20260928 23:44:05.763568 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.754903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.21.112.126:40301)
0928 23:44:05.755639 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.755657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.755696 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:44:05.756446 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.756465 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.756905 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.757356 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:05.757864 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:05.758938 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:05.759944 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:05.760457 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:05.760635 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:05.761704 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:05.761711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:05.761728 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:05.762159 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:05.762233 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:05.762258 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:05.762372 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:05.762505 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:44:05.762798 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":153,"threads_started":1}
I20260928 23:44:05.765655 27159 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b9837a871d4539bfe8196571484f13" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40301 } }
I20260928 23:44:05.765972 27159 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:05.766052 27159 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b9837a871d4539bfe8196571484f13, State: Initialized, Role: FOLLOWER
I20260928 23:44:05.766544 27159 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b9837a871d4539bfe8196571484f13" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40301 } }
I20260928 23:44:05.766717 27159 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:05.766795 27159 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:05.766912 27159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:05.766948 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:44:05.768205 27159 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b9837a871d4539bfe8196571484f13" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40301 } }
I20260928 23:44:05.768569 27159 leader_election.cc:302] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b9837a871d4539bfe8196571484f13; no voters:
I20260928 23:44:05.769017 27159 leader_election.cc:288] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:05.769179 27164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:05.769475 27164 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [term 1 LEADER]: Becoming Leader. State: Replica: f3b9837a871d4539bfe8196571484f13, State: Running, Role: LEADER
I20260928 23:44:05.769862 27164 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b9837a871d4539bfe8196571484f13" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40301 } }
I20260928 23:44:05.770447 27159 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:05.771816 27165 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b9837a871d4539bfe8196571484f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b9837a871d4539bfe8196571484f13" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40301 } } }
I20260928 23:44:05.771816 27166 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b9837a871d4539bfe8196571484f13. Latest consensus state: current_term: 1 leader_uuid: "f3b9837a871d4539bfe8196571484f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b9837a871d4539bfe8196571484f13" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40301 } } }
I20260928 23:44:05.772037 27165 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:05.772037 27166 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:05.772485 27176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:05.775647 27176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:05.782301 27176 catalog_manager.cc:1347] Generated new cluster ID: 86d48eeace4d435aa1fcac7203092102
I20260928 23:44:05.782397 27176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:05.797590 27176 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:05.798899 27176 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:05.803798 27176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b9837a871d4539bfe8196571484f13: Generated new TSK 0
I20260928 23:44:05.804721 27176 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:44:05.837903 27189 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 23:44:05.838110 27189 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:44:05.838650 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:05.820783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.21.112.126:40301)
0928 23:44:05.820929 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:05.820934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:05.821154 (+ 220us) server_negotiation.cc:407] Connection header received
0928 23:44:05.821382 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:05.821385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:05.821448 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:05.821599 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:44:05.821898 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:05.822765 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:05.823731 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:05.824091 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:05.824195 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:05.824565 (+ 370us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:44:05.838185 (+ 13620us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:44:05.838495 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:44:05.838748 27162 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47296: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 23:44:05.839885 27188 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:05.820639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40301 (local address 127.0.0.1:47296)
0928 23:44:05.821064 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:05.821083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:05.821204 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:05.821646 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:05.821648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:05.821661 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:44:05.821773 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:05.821787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:05.822892 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:05.822895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:05.823617 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:05.823625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:05.824232 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:05.824235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:05.824310 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:05.824359 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:44:05.838426 (+ 14067us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:44:05.838642 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.112.126:40301: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":131,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.112.126:40301: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 23:44:05.840695 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27092
2026-09-28T23:44:05Z chronyd exiting
I20260928 23:44:05.878182 21953 test_util.cc:175] -----------------------------------------------
I20260928 23:44:05.878296 21953 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790639014951155-21953-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 23:44:05.879408 21953 mini_oidc.cc:152] Starting JWKS server
I20260928 23:44:05.880285 21953 webserver.cc:460] Webserver started at https://127.0.0.1:34275/ using document root <none> and password file <none>
I20260928 23:44:05.880409 21953 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:44:05.880697 21953 webserver.cc:460] Webserver started at http://127.0.0.1:43301/ using document root <none> and password file <none>
2026-09-28T23:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:05Z Disabled control of system clock
I20260928 23:44:05.889982 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36475
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/testchainca.cert with env {}
W20260928 23:44:06.007956 27195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:06.008260 27195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:06.008301 27195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:06.014463 27195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:06.014575 27195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:06.014597 27195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:06.014639 27195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:06.014678 27195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:44:06.014708 27195 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:44:06.014743 27195 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34275/jwks.json
I20260928 23:44:06.022158 27195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:06.023365 27195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:06.024508 27195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:06.033758 27203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:06.033944 27195 server_base.cc:1034] running on GCE node
W20260928 23:44:06.033847 27200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:06.033761 27201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:06.034781 27195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:06.036012 27195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:06.037240 27195 hybrid_clock.cc:648] HybridClock initialized: now 1790639046037223 us; error 41 us; skew 500 ppm
I20260928 23:44:06.046624 27195 webserver.cc:460] Webserver started at http://127.21.112.126:40969/ using document root <none> and password file <none>
I20260928 23:44:06.047287 27195 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:06.047380 27195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:06.047622 27195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:06.049898 27195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "833d4c9c9f26440384984cdd4c29854c"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "26862606de7c4663eab71f3fd35817ac"
server_key_iv: "1bf647baa80106b1bdd7585ec3697b16"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.050486 27195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "833d4c9c9f26440384984cdd4c29854c"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "26862606de7c4663eab71f3fd35817ac"
server_key_iv: "1bf647baa80106b1bdd7585ec3697b16"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.054366 27195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:06.057009 27209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:06.058579 27195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:06.058722 27195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "833d4c9c9f26440384984cdd4c29854c"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "26862606de7c4663eab71f3fd35817ac"
server_key_iv: "1bf647baa80106b1bdd7585ec3697b16"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.058847 27195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:06.073000 27195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:06.073869 27195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:06.074074 27195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:06.081183 27195 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36475
I20260928 23:44:06.081190 27261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36475 every 8 connection(s)
I20260928 23:44:06.082250 27195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:06.085716 27262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:06.086894 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27195
I20260928 23:44:06.087054 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:06.087363 21953 external_mini_cluster.cc:1428] Setting key 0cac0c2cf4566c49c09d3515f9723d86
I20260928 23:44:06.092630 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c: Bootstrap starting.
I20260928 23:44:06.095554 27262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:06.096494 27262 log.cc:824] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:06.099433 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c: No bootstrap required, opened a new log
I20260928 23:44:06.100152 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:06.088883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.21.112.126:36475)
0928 23:44:06.089614 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:06.089633 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:06.089673 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:44:06.090405 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:06.090423 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:06.090847 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:06.091173 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:06.091550 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.092580 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.093819 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.094477 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.094678 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.096414 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:06.096425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:06.096448 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:06.097483 (+ 1035us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:06.097604 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:06.097640 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:06.097747 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:06.098013 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 23:44:06.098842 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260928 23:44:06.104974 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:44:06.106014 27262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833d4c9c9f26440384984cdd4c29854c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36475 } }
I20260928 23:44:06.106374 27262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:06.106455 27262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833d4c9c9f26440384984cdd4c29854c, State: Initialized, Role: FOLLOWER
I20260928 23:44:06.107158 27262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833d4c9c9f26440384984cdd4c29854c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36475 } }
I20260928 23:44:06.107324 27262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:06.107399 27262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:06.107509 27262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:06.108959 27262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833d4c9c9f26440384984cdd4c29854c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36475 } }
I20260928 23:44:06.109412 27262 leader_election.cc:302] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833d4c9c9f26440384984cdd4c29854c; no voters:
I20260928 23:44:06.110046 27262 leader_election.cc:288] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:06.110176 27267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:06.110419 27267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [term 1 LEADER]: Becoming Leader. State: Replica: 833d4c9c9f26440384984cdd4c29854c, State: Running, Role: LEADER
I20260928 23:44:06.110746 27267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833d4c9c9f26440384984cdd4c29854c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36475 } }
I20260928 23:44:06.111666 27262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:06.112617 27269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833d4c9c9f26440384984cdd4c29854c. Latest consensus state: current_term: 1 leader_uuid: "833d4c9c9f26440384984cdd4c29854c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833d4c9c9f26440384984cdd4c29854c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36475 } } }
I20260928 23:44:06.112552 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "833d4c9c9f26440384984cdd4c29854c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833d4c9c9f26440384984cdd4c29854c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36475 } } }
I20260928 23:44:06.112917 27269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:06.112917 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:06.113327 27276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:06.116648 27276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:06.124028 27276 catalog_manager.cc:1347] Generated new cluster ID: 4ffa909ec10443a3bd2d43d83bd9657b
I20260928 23:44:06.124203 27276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:06.149628 27276 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:06.150987 27276 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:06.159065 27276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 833d4c9c9f26440384984cdd4c29854c: Generated new TSK 0
I20260928 23:44:06.159987 27276 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:44:06.191007 27292 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 23:44:06.191114 27292 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:44:06.191848 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:06.175758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.21.112.126:36475)
0928 23:44:06.175904 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:06.175909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:06.176087 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:44:06.176255 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:06.176257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:06.176306 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:06.176400 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:44:06.176654 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.177198 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.178440 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.178679 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.178763 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.179161 (+ 398us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:44:06.191469 (+ 12308us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:44:06.191727 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 23:44:06.191928 27265 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 23:44:06.192101 27291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:06.175680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36475 (local address 127.0.0.1:37610)
0928 23:44:06.175994 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:06.176010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:06.176127 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:06.176422 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:06.176425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:06.176433 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:44:06.176545 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:06.176552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.177305 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.177308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:06.178320 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:06.178331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.178800 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.178804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:06.178900 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.178932 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:44:06.191696 (+ 12764us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:44:06.191899 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.112.126:36475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
I20260928 23:44:06.192713 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27195
2026-09-28T23:44:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (353 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 23:44:06.232481 21953 mini_oidc.cc:152] Starting JWKS server
I20260928 23:44:06.233372 21953 webserver.cc:460] Webserver started at https://127.0.0.1:34945/ using document root <none> and password file <none>
I20260928 23:44:06.233498 21953 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:44:06.233821 21953 webserver.cc:460] Webserver started at http://127.0.0.1:33853/ using document root <none> and password file <none>
2026-09-28T23:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:06Z Disabled control of system clock
I20260928 23:44:06.242689 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41581
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:36505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:41581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:44:06.356628 27298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:06.356912 27298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:06.356952 27298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:06.360945 27298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:06.361008 27298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:06.361024 27298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:06.361042 27298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:06.361056 27298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:44:06.361073 27298 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:44:06.361085 27298 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34945/jwks.json
I20260928 23:44:06.365983 27298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:41581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:06.367102 27298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:06.368455 27298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:06.377247 27306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:06.377221 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:06.377291 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:06.377753 27298 server_base.cc:1034] running on GCE node
I20260928 23:44:06.378341 27298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:06.379519 27298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:06.380690 27298 hybrid_clock.cc:648] HybridClock initialized: now 1790639046380672 us; error 32 us; skew 500 ppm
I20260928 23:44:06.383244 27298 webserver.cc:460] Webserver started at http://127.21.112.126:37371/ using document root <none> and password file <none>
I20260928 23:44:06.383893 27298 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:06.383963 27298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:06.384238 27298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:06.386488 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "6bf1a07d28d9410c97c74e8174c3c523"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "a872b2140c44ca038e27a76bf814b376"
server_key_iv: "0cf7ed5542c65f45a0e8b81b73782fdf"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.387063 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "6bf1a07d28d9410c97c74e8174c3c523"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "a872b2140c44ca038e27a76bf814b376"
server_key_iv: "0cf7ed5542c65f45a0e8b81b73782fdf"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.391006 27298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:06.393671 27312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:06.395023 27298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:06.395215 27298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "6bf1a07d28d9410c97c74e8174c3c523"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "a872b2140c44ca038e27a76bf814b376"
server_key_iv: "0cf7ed5542c65f45a0e8b81b73782fdf"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.395335 27298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:06.410048 27298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:06.410765 27298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:06.410962 27298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:06.418274 27298 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:41581
I20260928 23:44:06.418279 27364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:41581 every 8 connection(s)
I20260928 23:44:06.419263 27298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:06.422591 27365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:06.428270 27365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523: Bootstrap starting.
I20260928 23:44:06.428572 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27298
I20260928 23:44:06.428725 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:06.428985 21953 external_mini_cluster.cc:1428] Setting key 8258983e266ee029a40d8d41d23e995c
I20260928 23:44:06.431020 27365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:06.431767 27365 log.cc:824] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:06.434384 27365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523: No bootstrap required, opened a new log
I20260928 23:44:06.438176 27365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1a07d28d9410c97c74e8174c3c523" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41581 } }
I20260928 23:44:06.438438 27365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:06.438521 27365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf1a07d28d9410c97c74e8174c3c523, State: Initialized, Role: FOLLOWER
I20260928 23:44:06.438886 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:06.430326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41912 (local address 127.21.112.126:41581)
0928 23:44:06.431010 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:06.431024 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:06.431060 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:44:06.431792 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:06.431810 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:06.432330 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:06.432725 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:06.433231 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.434363 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.435362 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.435862 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.436037 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.436944 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:06.436949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:06.436963 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:06.437388 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:06.437448 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:06.437468 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:06.437619 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:06.437753 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 23:44:06.438031 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260928 23:44:06.439090 27365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1a07d28d9410c97c74e8174c3c523" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41581 } }
I20260928 23:44:06.439257 27365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:06.439324 27365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:06.439419 27365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:06.440485 27365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1a07d28d9410c97c74e8174c3c523" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41581 } }
I20260928 23:44:06.440902 27365 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf1a07d28d9410c97c74e8174c3c523; no voters:
I20260928 23:44:06.441313 27365 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:06.441483 27370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:06.441798 27370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf1a07d28d9410c97c74e8174c3c523, State: Running, Role: LEADER
I20260928 23:44:06.442046 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:44:06.442132 27370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1a07d28d9410c97c74e8174c3c523" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41581 } }
I20260928 23:44:06.442720 27365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:06.443264 27370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf1a07d28d9410c97c74e8174c3c523. Latest consensus state: current_term: 1 leader_uuid: "6bf1a07d28d9410c97c74e8174c3c523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1a07d28d9410c97c74e8174c3c523" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41581 } } }
I20260928 23:44:06.443383 27370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:06.443706 27371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf1a07d28d9410c97c74e8174c3c523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf1a07d28d9410c97c74e8174c3c523" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41581 } } }
I20260928 23:44:06.443782 27371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:06.444022 27376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:06.447289 27376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:06.453420 27376 catalog_manager.cc:1347] Generated new cluster ID: 6df5222cc71240a4b86658535e43f15f
I20260928 23:44:06.453506 27376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:06.476729 27376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:06.477849 27376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:06.484016 27376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf1a07d28d9410c97c74e8174c3c523: Generated new TSK 0
I20260928 23:44:06.484889 27376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:44:06.503690 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:06.498531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.21.112.126:41581)
0928 23:44:06.498679 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:06.498683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:06.498843 (+ 160us) server_negotiation.cc:407] Connection header received
0928 23:44:06.499006 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:06.499008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:06.499138 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:06.499264 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:44:06.499524 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.500167 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.501426 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.501697 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.501777 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.502092 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:44:06.503327 (+ 1235us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:44:06.503566 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 23:44:06.503774 27368 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 23:44:06.506577 27394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:06.498420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:41581 (local address 127.0.0.1:41920)
0928 23:44:06.498753 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:06.498768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:06.498882 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:06.499272 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:06.499275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:06.499284 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:44:06.499398 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:06.499405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.500292 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.500296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:06.501314 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:06.501325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.501810 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.501814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:06.501901 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.501931 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:44:06.503482 (+ 1551us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:44:06.506428 (+ 2946us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.112.126:41581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":2755,"thread_start_us":113,"threads_started":1}
I20260928 23:44:06.507280 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27298
E20260928 23:44:06.515599 27395 webserver.cc:533] Webserver: sslize failed:
E20260928 23:44:06.515677 27395 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T23:44:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T23:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:06Z Disabled control of system clock
I20260928 23:44:06.556340 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36487
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:40245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36487
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/testchainca.cert with env {}
W20260928 23:44:06.670246 27399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:06.670550 27399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:06.670593 27399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:06.674716 27399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:06.674786 27399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:06.674801 27399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:06.674818 27399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:06.674834 27399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:44:06.674850 27399 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:44:06.674865 27399 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 23:44:06.680469 27399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:40245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:06.681663 27399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:06.682718 27399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:06.691906 27404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:06.691982 27407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:06.691906 27405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:06.692162 27399 server_base.cc:1034] running on GCE node
I20260928 23:44:06.692961 27399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:06.694324 27399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:06.695545 27399 hybrid_clock.cc:648] HybridClock initialized: now 1790639046695499 us; error 52 us; skew 500 ppm
I20260928 23:44:06.697656 27399 webserver.cc:460] Webserver started at http://127.21.112.126:38323/ using document root <none> and password file <none>
I20260928 23:44:06.698261 27399 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:06.698350 27399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:06.698593 27399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:06.700740 27399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "38ab1095b4a34fc09dc851125c39a440"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "b3b56ee73184cbf83d10109c7756b7bc"
server_key_iv: "c4e211987241204d750a9f8051b5588c"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.701329 27399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "38ab1095b4a34fc09dc851125c39a440"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "b3b56ee73184cbf83d10109c7756b7bc"
server_key_iv: "c4e211987241204d750a9f8051b5588c"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.705186 27399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 23:44:06.707813 27413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:06.709194 27399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:06.709331 27399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "38ab1095b4a34fc09dc851125c39a440"
format_stamp: "Formatted at 2026-09-28 23:44:06 on dist-test-slave-c3s8"
server_key: "b3b56ee73184cbf83d10109c7756b7bc"
server_key_iv: "c4e211987241204d750a9f8051b5588c"
server_key_version: "encryptionkey@0"
I20260928 23:44:06.709447 27399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:06.736806 27399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:06.737681 27399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:06.737880 27399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:06.745373 27399 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36487
I20260928 23:44:06.745411 27465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36487 every 8 connection(s)
I20260928 23:44:06.746460 27399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:06.749711 27466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:06.752792 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27399
I20260928 23:44:06.752974 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:06.753262 21953 external_mini_cluster.cc:1428] Setting key 999f44cd1baee1d2173a3ab65d7c9d96
I20260928 23:44:06.756282 27466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440: Bootstrap starting.
I20260928 23:44:06.758771 27466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:06.759579 27466 log.cc:824] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:06.761618 27466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440: No bootstrap required, opened a new log
I20260928 23:44:06.762777 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:06.754706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.21.112.126:36487)
0928 23:44:06.755476 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:06.755493 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:06.755531 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:44:06.756154 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:06.756173 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:06.756638 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:06.756980 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:06.757482 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.758529 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.759467 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.759992 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.760159 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.761022 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:06.761028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:06.761048 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:06.761452 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:06.761508 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:06.761528 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:06.761692 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:06.761807 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 23:44:06.762081 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":144,"threads_started":1}
I20260928 23:44:06.765834 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:44:06.766108 27466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ab1095b4a34fc09dc851125c39a440" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36487 } }
I20260928 23:44:06.766351 27466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:06.766428 27466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ab1095b4a34fc09dc851125c39a440, State: Initialized, Role: FOLLOWER
I20260928 23:44:06.766952 27466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ab1095b4a34fc09dc851125c39a440" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36487 } }
I20260928 23:44:06.767122 27466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:06.767190 27466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:06.767287 27466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:06.768254 27466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ab1095b4a34fc09dc851125c39a440" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36487 } }
I20260928 23:44:06.768607 27466 leader_election.cc:302] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ab1095b4a34fc09dc851125c39a440; no voters:
I20260928 23:44:06.768918 27466 leader_election.cc:288] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:06.769042 27471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:06.769299 27471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [term 1 LEADER]: Becoming Leader. State: Replica: 38ab1095b4a34fc09dc851125c39a440, State: Running, Role: LEADER
I20260928 23:44:06.769654 27471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ab1095b4a34fc09dc851125c39a440" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36487 } }
I20260928 23:44:06.770256 27466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:06.771333 27473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ab1095b4a34fc09dc851125c39a440. Latest consensus state: current_term: 1 leader_uuid: "38ab1095b4a34fc09dc851125c39a440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ab1095b4a34fc09dc851125c39a440" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36487 } } }
I20260928 23:44:06.771469 27473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:06.771744 27472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ab1095b4a34fc09dc851125c39a440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ab1095b4a34fc09dc851125c39a440" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36487 } } }
I20260928 23:44:06.771847 27472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:06.771941 27480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:06.775278 27480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:06.781348 27480 catalog_manager.cc:1347] Generated new cluster ID: b4c11357fb8549db9f9991d580256b59
I20260928 23:44:06.781436 27480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:06.792228 27480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:06.793524 27480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:06.808252 27480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ab1095b4a34fc09dc851125c39a440: Generated new TSK 0
I20260928 23:44:06.809180 27480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:44:06.878456 27469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:06.816684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.21.112.126:36487)
0928 23:44:06.816816 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:06.816821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:06.817089 (+ 268us) server_negotiation.cc:407] Connection header received
0928 23:44:06.817319 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:06.817323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:06.817379 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:06.817492 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:44:06.817791 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.818384 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.819652 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.819873 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.819956 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.820338 (+ 382us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:44:06.877932 (+ 57594us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:44:06.878298 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57784: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 23:44:06.878551 27469 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57784: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 23:44:06.878585 27495 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:06.816560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36487 (local address 127.0.0.1:57784)
0928 23:44:06.816974 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:06.816989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:06.817125 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:06.817507 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:06.817510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:06.817518 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:44:06.817658 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:06.817665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.818498 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.818502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:06.819522 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:06.819532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:06.819995 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:06.820006 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:06.820106 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:06.820135 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:44:06.878194 (+ 58059us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:44:06.878379 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.112.126:36487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":115,"threads_started":1}
I20260928 23:44:06.879307 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27399
2026-09-28T23:44:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (351 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:07 dist-test-slave-c3s8 krb5kdc[27507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:07 dist-test-slave-c3s8 krb5kdc[27507](info): set up 2 sockets
Sep 28 23:44:07 dist-test-slave-c3s8 krb5kdc[27507](info): commencing operation
krb5kdc: starting...
W20260928 23:44:08.186769 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:08 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:44:08.322381 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:32829
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:08.435176 27520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:08.435475 27520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:08.435518 27520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:08.439841 27520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:08.439915 27520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:08.439935 27520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:08.439952 27520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:08.439968 27520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:08.445565 27520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:08.446770 27520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:08.447929 27520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:08.457017 27528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:08.456964 27526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:08.456964 27525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:08.457223 27520 server_base.cc:1034] running on GCE node
I20260928 23:44:08.457976 27520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:08.459334 27520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:08.460510 27520 hybrid_clock.cc:648] HybridClock initialized: now 1790639048460480 us; error 45 us; skew 500 ppm
Sep 28 23:44:08 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639048, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:08.468353 27520 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:44:08.469506 27520 webserver.cc:460] Webserver started at http://127.21.112.126:35189/ using document root <none> and password file <none>
I20260928 23:44:08.470163 27520 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:08.470258 27520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:08.470499 27520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:08.472640 27520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "d92765c69d0a4563af12d227910764b9"
format_stamp: "Formatted at 2026-09-28 23:44:08 on dist-test-slave-c3s8"
server_key: "beff40ba2b8038d2dd085d15657dcce0"
server_key_iv: "3f06d9a35786d7362f6b180597df2466"
server_key_version: "encryptionkey@0"
I20260928 23:44:08.473234 27520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "d92765c69d0a4563af12d227910764b9"
format_stamp: "Formatted at 2026-09-28 23:44:08 on dist-test-slave-c3s8"
server_key: "beff40ba2b8038d2dd085d15657dcce0"
server_key_iv: "3f06d9a35786d7362f6b180597df2466"
server_key_version: "encryptionkey@0"
I20260928 23:44:08.477820 27520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 23:44:08.480484 27535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:08.481945 27520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:08.482093 27520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "d92765c69d0a4563af12d227910764b9"
format_stamp: "Formatted at 2026-09-28 23:44:08 on dist-test-slave-c3s8"
server_key: "beff40ba2b8038d2dd085d15657dcce0"
server_key_iv: "3f06d9a35786d7362f6b180597df2466"
server_key_version: "encryptionkey@0"
I20260928 23:44:08.482220 27520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:08.502152 27520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:08.505566 27520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:08.505772 27520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:08.513063 27520 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:32829
I20260928 23:44:08.513087 27587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:32829 every 8 connection(s)
I20260928 23:44:08.514117 27520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:08.517042 27588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:08.519479 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27520
I20260928 23:44:08.519636 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:08.519917 21953 external_mini_cluster.cc:1428] Setting key 94d56a9001aa12f8f722773f4f57e6ca
I20260928 23:44:08.523559 27588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9: Bootstrap starting.
I20260928 23:44:08.525949 27588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:08.526722 27588 log.cc:824] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:08.528878 27588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9: No bootstrap required, opened a new log
Sep 28 23:44:08 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:08.531996 27588 raft_consensus.cc:348] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92765c69d0a4563af12d227910764b9" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 32829 } }
I20260928 23:44:08.532228 27588 raft_consensus.cc:374] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:08.532262 27588 raft_consensus.cc:729] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92765c69d0a4563af12d227910764b9, State: Initialized, Role: FOLLOWER
I20260928 23:44:08.532756 27588 consensus_queue.cc:260] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92765c69d0a4563af12d227910764b9" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 32829 } }
I20260928 23:44:08.532879 27588 raft_consensus.cc:388] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:08.533010 27588 raft_consensus.cc:482] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:08.533150 27588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:08.534161 27588 raft_consensus.cc:504] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92765c69d0a4563af12d227910764b9" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 32829 } }
I20260928 23:44:08.534574 27588 leader_election.cc:302] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d92765c69d0a4563af12d227910764b9; no voters:
I20260928 23:44:08.534885 27588 leader_election.cc:288] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:08.534991 27593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:08.535219 27593 raft_consensus.cc:686] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [term 1 LEADER]: Becoming Leader. State: Replica: d92765c69d0a4563af12d227910764b9, State: Running, Role: LEADER
I20260928 23:44:08.535473 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:08.521417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.21.112.126:32829)
0928 23:44:08.521913 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:08.521928 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:08.521966 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:44:08.522681 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:08.522700 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:08.523053 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:08.523410 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:08.524225 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:08.525240 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:08.526192 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:08.526695 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:08.526871 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:08.529635 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:08.529641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:08.529655 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:08.529684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:08.531940 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:08.532518 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:08.532523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:08.532528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:08.532644 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:08.532973 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:08.532977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:08.532979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:08.533521 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:08.533776 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:08.534115 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 23:44:08.534441 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":151,"threads_started":1}
I20260928 23:44:08.535597 27593 consensus_queue.cc:237] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92765c69d0a4563af12d227910764b9" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 32829 } }
I20260928 23:44:08.536269 27588 sys_catalog.cc:564] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:08.536819 27593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d92765c69d0a4563af12d227910764b9. Latest consensus state: current_term: 1 leader_uuid: "d92765c69d0a4563af12d227910764b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92765c69d0a4563af12d227910764b9" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 32829 } } }
I20260928 23:44:08.536935 27593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:08.537626 27594 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d92765c69d0a4563af12d227910764b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92765c69d0a4563af12d227910764b9" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 32829 } } }
I20260928 23:44:08.537726 27594 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:08.537987 27598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:08.541265 27598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:08.547775 27598 catalog_manager.cc:1347] Generated new cluster ID: 4a0c19ee5113417a9b632b8a22de83d5
I20260928 23:44:08.547865 27598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:08.576788 27598 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:08.578343 27598 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:08.589699 27598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d92765c69d0a4563af12d227910764b9: Generated new TSK 0
I20260928 23:44:08.590593 27598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:44:08.628875 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:32829
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:08.747731 27614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:08.748050 27614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:08.748096 27614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:08.752540 27614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:08.752627 27614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:08.752691 27614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:08.758901 27614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:08.760427 27614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:08.761763 27614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:08.774020 27619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:08.773988 27622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:08.774047 27620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:08.774549 27614 server_base.cc:1034] running on GCE node
I20260928 23:44:08.775218 27614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:08.776034 27614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:08.777252 27614 hybrid_clock.cc:648] HybridClock initialized: now 1790639048777218 us; error 51 us; skew 500 ppm
Sep 28 23:44:08 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639048, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:08.780711 27614 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:44:08.781783 27614 webserver.cc:460] Webserver started at http://127.21.112.65:39935/ using document root <none> and password file <none>
I20260928 23:44:08.782429 27614 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:08.782497 27614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:08.782719 27614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:08.785012 27614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "6e6810d87ae3425680634fe6a6214deb"
format_stamp: "Formatted at 2026-09-28 23:44:08 on dist-test-slave-c3s8"
server_key: "e74b0f5cd3137946e23d73dd447bdb85"
server_key_iv: "30cf75b17d5e04051bc8cd1e0152885d"
server_key_version: "encryptionkey@0"
I20260928 23:44:08.785766 27614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "6e6810d87ae3425680634fe6a6214deb"
format_stamp: "Formatted at 2026-09-28 23:44:08 on dist-test-slave-c3s8"
server_key: "e74b0f5cd3137946e23d73dd447bdb85"
server_key_iv: "30cf75b17d5e04051bc8cd1e0152885d"
server_key_version: "encryptionkey@0"
I20260928 23:44:08.790273 27614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:44:08.793341 27629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:08.794839 27614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:08.795013 27614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "6e6810d87ae3425680634fe6a6214deb"
format_stamp: "Formatted at 2026-09-28 23:44:08 on dist-test-slave-c3s8"
server_key: "e74b0f5cd3137946e23d73dd447bdb85"
server_key_iv: "30cf75b17d5e04051bc8cd1e0152885d"
server_key_version: "encryptionkey@0"
I20260928 23:44:08.795168 27614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:08.818164 27614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:08.821686 27614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:08.821897 27614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:08.822556 27614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:08.823410 27614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:08.823487 27614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:08.823558 27614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:08.823611 27614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:08.832851 27614 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:32803
I20260928 23:44:08.832872 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:32803 every 8 connection(s)
I20260928 23:44:08.833804 27614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:08.836133 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27614
I20260928 23:44:08.836299 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:08.836668 21953 external_mini_cluster.cc:1428] Setting key cd612576f939536cc81759f76e51f1af
Sep 28 23:44:08 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639048, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:44:08.851804 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:08.836049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45115 (local address 127.21.112.126:32829)
0928 23:44:08.836228 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:08.836233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:08.837379 (+ 1146us) server_negotiation.cc:407] Connection header received
0928 23:44:08.839443 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:08.839445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:08.839490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:08.839556 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:08.841083 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:08.841846 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:08.843937 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:08.844175 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:08.844266 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:08.847383 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:08.847388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:08.847393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:08.847431 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:08.849213 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:08.849830 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:08.849834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:08.849837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:08.849926 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:08.850379 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:08.850382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:08.850384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:08.850634 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:08.850745 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:08.851453 (+ 708us) server_negotiation.cc:299] Negotiation successful
0928 23:44:08.851603 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:44:08.852864 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:08.836571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:32829 (local address 127.21.112.65:45115)
0928 23:44:08.837162 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:08.837257 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:08.839177 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:08.839757 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:08.839771 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:08.840364 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:08.840916 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:08.840931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:08.842052 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:08.842060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:08.843793 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:08.843810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:08.844356 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:08.844363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:08.844639 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:08.845335 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:08.845361 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:08.847186 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:08.849346 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:08.849352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:08.849369 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:08.849722 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:08.850092 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:08.850098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:08.850100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:08.850272 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:08.850746 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:08.850754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:08.851101 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 23:44:08.851393 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 23:44:08.851793 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":162,"threads_started":1}
I20260928 23:44:08.854342 27743 heartbeater.cc:347] Connected to a master server at 127.21.112.126:32829
I20260928 23:44:08.854635 27743 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:08.855186 27743 heartbeater.cc:511] Master 127.21.112.126:32829 requested a full tablet report, sending...
I20260928 23:44:08.856837 27550 ts_manager.cc:194] Registered new tserver with Master: 6e6810d87ae3425680634fe6a6214deb (127.21.112.65:32803)
I20260928 23:44:08.858178 27550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:45115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:44:08.869671 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:32829
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:08.982389 27748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:08.982663 27748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:08.982707 27748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:08.986975 27748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:08.987046 27748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:08.987112 27748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:08.992491 27748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:08.993654 27748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:08.994689 27748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:09.005061 27754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:09.005257 27748 server_base.cc:1034] running on GCE node
W20260928 23:44:09.005066 27756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:09.005098 27753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:09.005950 27748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:09.006750 27748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:09.007957 27748 hybrid_clock.cc:648] HybridClock initialized: now 1790639049007931 us; error 39 us; skew 500 ppm
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.011191 27748 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:44:09.012228 27748 webserver.cc:460] Webserver started at http://127.21.112.66:34385/ using document root <none> and password file <none>
I20260928 23:44:09.012853 27748 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:09.012955 27748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:09.013206 27748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:09.015467 27748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "7f534ec510db406ab16fa41e28e7f85e"
format_stamp: "Formatted at 2026-09-28 23:44:09 on dist-test-slave-c3s8"
server_key: "8cd1f6c1598ab8b092286efa83b71d16"
server_key_iv: "a0f02ca3e8ed0a49fc596caee35bec93"
server_key_version: "encryptionkey@0"
I20260928 23:44:09.016065 27748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "7f534ec510db406ab16fa41e28e7f85e"
format_stamp: "Formatted at 2026-09-28 23:44:09 on dist-test-slave-c3s8"
server_key: "8cd1f6c1598ab8b092286efa83b71d16"
server_key_iv: "a0f02ca3e8ed0a49fc596caee35bec93"
server_key_version: "encryptionkey@0"
I20260928 23:44:09.020251 27748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:09.023221 27763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:09.024672 27748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:44:09.024839 27748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "7f534ec510db406ab16fa41e28e7f85e"
format_stamp: "Formatted at 2026-09-28 23:44:09 on dist-test-slave-c3s8"
server_key: "8cd1f6c1598ab8b092286efa83b71d16"
server_key_iv: "a0f02ca3e8ed0a49fc596caee35bec93"
server_key_version: "encryptionkey@0"
I20260928 23:44:09.024972 27748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:09.040904 27748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:09.044179 27748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:09.044395 27748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:09.045051 27748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:09.045986 27748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:09.046056 27748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:09.046137 27748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:09.046187 27748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:09.055774 27748 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:36669
I20260928 23:44:09.055846 27876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:36669 every 8 connection(s)
I20260928 23:44:09.056771 27748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:09.066692 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27748
I20260928 23:44:09.066844 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:09.067278 21953 external_mini_cluster.cc:1428] Setting key a6fbdceb73a0929ab80244d0a99d373c
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:09.073271 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.058899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:41817 (local address 127.21.112.126:32829)
0928 23:44:09.059062 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.059066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.060108 (+ 1042us) server_negotiation.cc:407] Connection header received
0928 23:44:09.061725 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.061728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.061794 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.061914 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.063277 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.063985 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.065205 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.065458 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.065629 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.068423 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:09.068427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:09.068429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:09.068454 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:09.070422 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.071013 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.071019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.071023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.071137 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.071656 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.071660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.071661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.071878 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:09.071976 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:09.072741 (+ 765us) server_negotiation.cc:299] Negotiation successful
0928 23:44:09.072920 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:44:09.073961 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.059330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:32829 (local address 127.21.112.66:41817)
0928 23:44:09.059841 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:09.059977 (+ 136us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:09.061419 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:09.062146 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:09.062154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:09.062569 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:09.063111 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:09.063126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.064124 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.064128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:09.065031 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:09.065040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.065669 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.065672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:09.065820 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.066427 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:09.066443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:09.068229 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:09.070556 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:09.070562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:09.070579 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:09.070887 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:09.071260 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:09.071265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:09.071268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:09.071499 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:09.072000 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:09.072009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:09.072414 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 23:44:09.072644 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 23:44:09.072961 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260928 23:44:09.075495 27877 heartbeater.cc:347] Connected to a master server at 127.21.112.126:32829
I20260928 23:44:09.075768 27877 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:09.076385 27877 heartbeater.cc:511] Master 127.21.112.126:32829 requested a full tablet report, sending...
I20260928 23:44:09.077730 27550 ts_manager.cc:194] Registered new tserver with Master: 7f534ec510db406ab16fa41e28e7f85e (127.21.112.66:36669)
I20260928 23:44:09.078374 27550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:41817
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:44:09.100804 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:32829
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:09.217898 27882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:09.218181 27882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:09.218226 27882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:09.222349 27882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:09.222415 27882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:09.222476 27882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:09.227861 27882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:32829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:09.228997 27882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:09.230214 27882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:09.240484 27888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:09.240633 27882 server_base.cc:1034] running on GCE node
W20260928 23:44:09.240535 27890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:09.240501 27887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:09.241345 27882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:09.242058 27882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:09.243251 27882 hybrid_clock.cc:648] HybridClock initialized: now 1790639049243224 us; error 42 us; skew 500 ppm
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.246483 27882 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:44:09.247651 27882 webserver.cc:460] Webserver started at http://127.21.112.67:44699/ using document root <none> and password file <none>
I20260928 23:44:09.248270 27882 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:09.248364 27882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:09.248600 27882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:09.250808 27882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "086f5dd8c0c54083aa72ce7bd17feea1"
format_stamp: "Formatted at 2026-09-28 23:44:09 on dist-test-slave-c3s8"
server_key: "9cd189d7ef7e0eab9d6672f2d9f04aa4"
server_key_iv: "351c17508eb63f9855fb803f5fbead21"
server_key_version: "encryptionkey@0"
I20260928 23:44:09.251394 27882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "086f5dd8c0c54083aa72ce7bd17feea1"
format_stamp: "Formatted at 2026-09-28 23:44:09 on dist-test-slave-c3s8"
server_key: "9cd189d7ef7e0eab9d6672f2d9f04aa4"
server_key_iv: "351c17508eb63f9855fb803f5fbead21"
server_key_version: "encryptionkey@0"
I20260928 23:44:09.255686 27882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:44:09.258667 27897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:09.260183 27882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:09.260349 27882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "086f5dd8c0c54083aa72ce7bd17feea1"
format_stamp: "Formatted at 2026-09-28 23:44:09 on dist-test-slave-c3s8"
server_key: "9cd189d7ef7e0eab9d6672f2d9f04aa4"
server_key_iv: "351c17508eb63f9855fb803f5fbead21"
server_key_version: "encryptionkey@0"
I20260928 23:44:09.260516 27882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:09.274601 27882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:09.278337 27882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:09.278509 27882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:09.279091 27882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:09.279933 27882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:09.279992 27882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:09.280032 27882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:09.280047 27882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:09.289904 27882 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:38789
I20260928 23:44:09.289947 28010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:38789 every 8 connection(s)
I20260928 23:44:09.290974 27882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:09.298287 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 27882
I20260928 23:44:09.298465 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:09.298759 21953 external_mini_cluster.cc:1428] Setting key b6fba3fdc5542481b74c58d8f3da608e
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:09.306154 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.293068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:52023 (local address 127.21.112.126:32829)
0928 23:44:09.293252 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.293256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.294202 (+ 946us) server_negotiation.cc:407] Connection header received
0928 23:44:09.295722 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.295725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.295778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.295861 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.297146 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.297773 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.299315 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.299608 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.299759 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.302527 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:09.302529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:09.302531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:09.302559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:09.304043 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.304561 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.304563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.304565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.304645 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.304981 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.304984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.304985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.305199 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:09.305289 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:09.305888 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 23:44:09.306017 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:44:09.306998 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.293475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:32829 (local address 127.21.112.67:52023)
0928 23:44:09.294048 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:09.294093 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:09.295479 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:09.296106 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:09.296113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:09.296495 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:09.296987 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:09.297001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.297907 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.297910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:09.299148 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:09.299160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.299813 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.299819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:09.299989 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.300679 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:09.300700 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:09.302367 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:09.304194 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:09.304199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:09.304213 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:09.304475 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:09.304720 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:09.304722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:09.304723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:09.304856 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:09.305304 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:09.305310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:09.305640 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 23:44:09.305821 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 23:44:09.306099 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":123,"threads_started":1}
I20260928 23:44:09.308303 28011 heartbeater.cc:347] Connected to a master server at 127.21.112.126:32829
I20260928 23:44:09.308535 28011 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:09.309037 28011 heartbeater.cc:511] Master 127.21.112.126:32829 requested a full tablet report, sending...
I20260928 23:44:09.310174 27550 ts_manager.cc:194] Registered new tserver with Master: 086f5dd8c0c54083aa72ce7bd17feea1 (127.21.112.67:38789)
I20260928 23:44:09.310727 27550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:52023
I20260928 23:44:09.312978 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/test-admin.keytab.
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.331429 21953 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 23:44:09.335124 28018 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:09.333135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.21.112.65:32803)
0928 23:44:09.333454 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.333477 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.333561 (+ 84us) server_negotiation.cc:407] Connection header received
0928 23:44:09.333773 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.333781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.334028 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.334199 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.334776 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39418: BlockingRecv error: recv got EOF from 127.0.0.1:39418 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
W20260928 23:44:09.335139 21953 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.336355 21953 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.337469 21953 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:44:09.348150 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.338580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.21.112.65:32803)
0928 23:44:09.338714 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.338718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.338838 (+ 120us) server_negotiation.cc:407] Connection header received
0928 23:44:09.339046 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.339049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.339100 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.339188 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.339749 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.340554 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.341398 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.341887 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.342014 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.344306 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:09.344316 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:09.344339 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:09.344364 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:09.346231 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.346723 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.346729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.346730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.346810 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.347112 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.347116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.347118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.347379 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:09.347605 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:09.347826 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 23:44:09.347979 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 23:44:09.352396 28018 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:09.351402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.21.112.65:32803)
0928 23:44:09.351545 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.351549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.351563 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:09.351653 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.351655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.351698 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.351776 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.352232 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39444: BlockingRecv error: recv got EOF from 127.0.0.1:39444 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 23:44:09.352527 21953 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.353718 21953 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.354663 21953 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:44:09.364329 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.355566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.21.112.65:32803)
0928 23:44:09.355682 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.355685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.355700 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:09.355816 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.355818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.355852 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.355916 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.356531 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.357115 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.357981 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.358273 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.358362 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.360691 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:09.360694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:09.360696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:09.360719 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:09.362437 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.362851 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.362854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.362856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.363006 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.363308 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.363311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.363312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.363505 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:09.363614 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:09.363849 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 23:44:09.364204 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 23:44:09.367597 28018 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:09.366373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.21.112.65:32803)
0928 23:44:09.366515 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.366520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.366627 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:44:09.366793 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.366796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.366839 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.366940 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.367426 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39466: BlockingRecv error: recv got EOF from 127.0.0.1:39466 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 23:44:09.367959 21953 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790639014951155-21953-0/krb5kdc/krb5cc)
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.369066 21953 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:09.370375 21953 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[27507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:44:09.379830 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:09.371270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.21.112.65:32803)
0928 23:44:09.371399 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:09.371404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:09.371563 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:44:09.371721 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:09.371724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:09.371759 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:09.371840 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:09.372335 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.372832 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.373781 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:09.374005 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:09.374116 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:09.376459 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:09.376463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:09.376465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:09.376489 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:09.378065 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.378526 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.378531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.378533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.378619 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:09.378940 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:09.378944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:09.378946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:09.379136 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:09.379283 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:09.379543 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 23:44:09.379658 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:09.381013 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27614
I20260928 23:44:09.387017 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27748
I20260928 23:44:09.392457 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27882
I20260928 23:44:09.398401 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 27520
2026-09-28T23:44:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2268 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[28031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[28031](info): set up 2 sockets
Sep 28 23:44:09 dist-test-slave-c3s8 krb5kdc[28031](info): commencing operation
krb5kdc: starting...
W20260928 23:44:10.454617 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:10 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab.
Entry for principal oryx/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab.
I20260928 23:44:10.585309 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46615
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab
--principal=oryx/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:10.701138 28044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:10.701468 28044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:10.701603 28044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:10.705740 28044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:10.705839 28044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:10.705880 28044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:10.705917 28044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:10.705956 28044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:10.711458 28044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab
--principal=oryx/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:10.712566 28044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:10.713661 28044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:10.722334 28052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:10.722352 28050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:10.722337 28049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:10.722792 28044 server_base.cc:1034] running on GCE node
I20260928 23:44:10.723415 28044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:10.724620 28044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:10.725816 28044 hybrid_clock.cc:648] HybridClock initialized: now 1790639050725795 us; error 35 us; skew 500 ppm
Sep 28 23:44:10 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:10.728737 28044 init.cc:377] Logged in from keytab as oryx/127.21.112.126@KRBTEST.COM (short username oryx)
I20260928 23:44:10.729774 28044 webserver.cc:460] Webserver started at http://127.21.112.126:42727/ using document root <none> and password file <none>
I20260928 23:44:10.730350 28044 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:10.730444 28044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:10.730684 28044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:10.732820 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "f1b0b16080d14035b8611d345c843120"
format_stamp: "Formatted at 2026-09-28 23:44:10 on dist-test-slave-c3s8"
server_key: "cd5e56d2f22431aa103e8913e2732d38"
server_key_iv: "c3f99740488dc1d7b542bdabbaae3bd4"
server_key_version: "encryptionkey@0"
I20260928 23:44:10.733415 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "f1b0b16080d14035b8611d345c843120"
format_stamp: "Formatted at 2026-09-28 23:44:10 on dist-test-slave-c3s8"
server_key: "cd5e56d2f22431aa103e8913e2732d38"
server_key_iv: "c3f99740488dc1d7b542bdabbaae3bd4"
server_key_version: "encryptionkey@0"
I20260928 23:44:10.737334 28044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:44:10.739936 28059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:10.741302 28044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:10.741465 28044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "f1b0b16080d14035b8611d345c843120"
format_stamp: "Formatted at 2026-09-28 23:44:10 on dist-test-slave-c3s8"
server_key: "cd5e56d2f22431aa103e8913e2732d38"
server_key_iv: "c3f99740488dc1d7b542bdabbaae3bd4"
server_key_version: "encryptionkey@0"
I20260928 23:44:10.741659 28044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:10.755218 28044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:10.759207 28044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:10.759454 28044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:10.767221 28044 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:46615
I20260928 23:44:10.767573 28111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:46615 every 8 connection(s)
I20260928 23:44:10.768321 28044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:10.771479 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:10.772495 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28044
I20260928 23:44:10.772643 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:10.772995 21953 external_mini_cluster.cc:1428] Setting key e7747cf8d80e1b803a14a339c8590712
I20260928 23:44:10.778659 28112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Bootstrap starting.
I20260928 23:44:10.781241 28112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:10.782155 28112 log.cc:824] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:44:10 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:10.784636 28112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: No bootstrap required, opened a new log
I20260928 23:44:10.788347 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:10.774650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.21.112.126:46615)
0928 23:44:10.775341 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:10.775357 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:10.775396 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:44:10.776157 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:10.776178 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:10.776540 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:10.777016 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:10.777942 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:10.779020 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:10.779917 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:10.780427 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:10.780605 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:10.783593 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:10.783598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:10.783613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:10.783648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:10.785422 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:10.785923 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:10.785929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:10.785933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:10.786046 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:10.786349 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:10.786352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:10.786354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:10.786786 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:10.786989 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:10.787282 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 23:44:10.787579 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":150,"threads_started":1}
I20260928 23:44:10.788482 28112 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } }
I20260928 23:44:10.788785 28112 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:10.788867 28112 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b0b16080d14035b8611d345c843120, State: Initialized, Role: FOLLOWER
I20260928 23:44:10.789386 28112 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } }
I20260928 23:44:10.789575 28112 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:10.789650 28112 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:10.789752 28112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:10.790824 28112 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } }
I20260928 23:44:10.791211 28112 leader_election.cc:302] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b0b16080d14035b8611d345c843120; no voters:
I20260928 23:44:10.791532 28112 leader_election.cc:288] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:10.791823 28117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:10.792079 28117 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 LEADER]: Becoming Leader. State: Replica: f1b0b16080d14035b8611d345c843120, State: Running, Role: LEADER
I20260928 23:44:10.792394 28117 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } }
I20260928 23:44:10.792884 28112 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:10.793632 28117 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b0b16080d14035b8611d345c843120. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } }
I20260928 23:44:10.794281 28118 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } }
I20260928 23:44:10.794404 28117 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:10.794582 28118 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
W20260928 23:44:10.798203 28132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:44:10.798295 28132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:44:10.798352 28129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:10.799192 28129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:10.805244 28129 catalog_manager.cc:1347] Generated new cluster ID: 9a93cd82a70147caaac56fd3a8a59bcf
I20260928 23:44:10.805331 28129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:10.815636 28129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:10.816728 28129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:10.825686 28129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Generated new TSK 0
I20260928 23:44:10.826538 28129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab.
Entry for principal oryx/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab.
I20260928 23:44:10.869798 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46615
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:10.989643 28138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:10.989956 28138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:10.990001 28138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:10.994374 28138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:10.994446 28138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:10.994513 28138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:11.000236 28138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:11.001353 28138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:11.002372 28138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:11.012342 28143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.012351 28144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:11.012647 28138 server_base.cc:1034] running on GCE node
W20260928 23:44:11.012686 28146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:11.013195 28138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:11.014050 28138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:11.015224 28138 hybrid_clock.cc:648] HybridClock initialized: now 1790639051015207 us; error 31 us; skew 500 ppm
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:11.018263 28138 init.cc:377] Logged in from keytab as oryx/127.21.112.65@KRBTEST.COM (short username oryx)
I20260928 23:44:11.019194 28138 webserver.cc:460] Webserver started at http://127.21.112.65:37437/ using document root <none> and password file <none>
I20260928 23:44:11.019762 28138 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:11.019824 28138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:11.020078 28138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:11.022403 28138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "f87a2836595e4202906b910e99da3dd1"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c6d27707d28853b383f5c3cd03cecf59"
server_key_iv: "ebaf371a33e418645cf4b21cbc18de29"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.022979 28138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "f87a2836595e4202906b910e99da3dd1"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c6d27707d28853b383f5c3cd03cecf59"
server_key_iv: "ebaf371a33e418645cf4b21cbc18de29"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.027017 28138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:11.029805 28153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.031123 28138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:11.031271 28138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "f87a2836595e4202906b910e99da3dd1"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c6d27707d28853b383f5c3cd03cecf59"
server_key_iv: "ebaf371a33e418645cf4b21cbc18de29"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.031394 28138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:11.051946 28138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:11.055297 28138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:11.055531 28138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:11.056188 28138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:11.057169 28138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:11.057253 28138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.057334 28138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:11.057390 28138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.067252 28138 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:46149
I20260928 23:44:11.067278 28266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:46149 every 8 connection(s)
I20260928 23:44:11.068459 28138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:11.077433 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28138
I20260928 23:44:11.077616 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:11.077996 21953 external_mini_cluster.cc:1428] Setting key ecf85d2df8a27999a9dfe9e729e4e573
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:11.084107 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.070632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:51923 (local address 127.21.112.126:46615)
0928 23:44:11.070815 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.070819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.071804 (+ 985us) server_negotiation.cc:407] Connection header received
0928 23:44:11.073303 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.073306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.073364 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.073482 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.074710 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.075252 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.076442 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.076687 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.076801 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.080015 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.080019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.080021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.080046 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.081761 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.082363 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.082368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.082370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.082460 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.082863 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.082866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.082868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.083076 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.083189 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.083775 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.083926 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 23:44:11.085098 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.071113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:51923)
0928 23:44:11.071631 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.071676 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.073082 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.073689 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.073696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.074072 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.074560 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.074574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.075386 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.075390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.076315 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.076324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.076845 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.076849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.077036 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.077856 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.077879 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.079813 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.081912 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.081919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.081938 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.082245 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.082562 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.082568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.082574 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.082748 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.083201 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.083211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.083497 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.083732 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.084127 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":131,"threads_started":1}
I20260928 23:44:11.086575 28267 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46615
I20260928 23:44:11.086856 28267 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:44:11.087460 28267 heartbeater.cc:511] Master 127.21.112.126:46615 requested a full tablet report, sending...
I20260928 23:44:11.089253 28076 ts_manager.cc:194] Registered new tserver with Master: f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.66@KRBTEST.COM" created.
I20260928 23:44:11.090693 28076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:51923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab.
Entry for principal oryx/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab.
I20260928 23:44:11.109736 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46615
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:11.222700 28272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:11.223012 28272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:11.223096 28272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:11.227186 28272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:11.227281 28272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:11.227387 28272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:11.232760 28272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:11.234062 28272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:11.235172 28272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:11.244870 28277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.244884 28280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.244896 28278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:11.244993 28272 server_base.cc:1034] running on GCE node
I20260928 23:44:11.245680 28272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:11.246372 28272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:11.247625 28272 hybrid_clock.cc:648] HybridClock initialized: now 1790639051247606 us; error 34 us; skew 500 ppm
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:11.250897 28272 init.cc:377] Logged in from keytab as oryx/127.21.112.66@KRBTEST.COM (short username oryx)
I20260928 23:44:11.251904 28272 webserver.cc:460] Webserver started at http://127.21.112.66:38731/ using document root <none> and password file <none>
I20260928 23:44:11.252563 28272 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:11.252632 28272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:11.252849 28272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:11.255155 28272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "68d2645e4057458692cde93aaed70e46"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c4ee9fbc25ac16a091fcd1661006724b"
server_key_iv: "a0e8ed14a291b332bd328b165cb220a3"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.255728 28272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "68d2645e4057458692cde93aaed70e46"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c4ee9fbc25ac16a091fcd1661006724b"
server_key_iv: "a0e8ed14a291b332bd328b165cb220a3"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.259729 28272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:11.262462 28287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.263778 28272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:11.263906 28272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "68d2645e4057458692cde93aaed70e46"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c4ee9fbc25ac16a091fcd1661006724b"
server_key_iv: "a0e8ed14a291b332bd328b165cb220a3"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.264015 28272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:11.282721 28272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:11.286541 28272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:11.286761 28272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:11.287460 28272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:11.288596 28272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:11.288651 28272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.288692 28272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:11.288707 28272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.298728 28272 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:46087
I20260928 23:44:11.298832 28400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:46087 every 8 connection(s)
I20260928 23:44:11.299681 28272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:11.306739 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28272
I20260928 23:44:11.306881 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:11.307225 21953 external_mini_cluster.cc:1428] Setting key eec4b5960f863c8abbd6fb4c3a2c5861
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:11.315557 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.301596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:40265 (local address 127.21.112.126:46615)
0928 23:44:11.301738 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.301742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.302628 (+ 886us) server_negotiation.cc:407] Connection header received
0928 23:44:11.304053 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.304056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.304104 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.304181 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.305447 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.306122 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.307432 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.307782 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.307897 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.311222 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.311227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.311229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.311254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.312937 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.313523 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.313530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.313557 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.313659 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.314054 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.314058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.314060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.314269 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.314402 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.315159 (+ 757us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.315337 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:44:11.316759 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.302011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:40265)
0928 23:44:11.302462 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.302505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.303829 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.304337 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.304352 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.304707 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.305266 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.305282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.306251 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.306254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.307271 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.307282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.307958 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.307961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.308173 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.308981 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.308998 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.311002 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.313086 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.313094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.313112 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.313405 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.313768 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.313772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.313775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.313938 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.314426 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.314437 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.314826 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.315089 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.315503 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:44:11.318114 28401 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46615
I20260928 23:44:11.318394 28401 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:11.318941 28401 heartbeater.cc:511] Master 127.21.112.126:46615 requested a full tablet report, sending...
I20260928 23:44:11.320119 28076 ts_manager.cc:194] Registered new tserver with Master: 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.67@KRBTEST.COM" created.
I20260928 23:44:11.320791 28076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:40265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab.
Entry for principal oryx/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab.
I20260928 23:44:11.338191 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46615
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:11.450657 28406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:11.450945 28406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:11.450989 28406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:11.455166 28406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:11.455231 28406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:11.455294 28406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:11.460704 28406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:11.461854 28406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:11.462909 28406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:11.473198 28412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.473289 28411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.473212 28414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:11.473727 28406 server_base.cc:1034] running on GCE node
I20260928 23:44:11.474159 28406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:11.474812 28406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:11.476073 28406 hybrid_clock.cc:648] HybridClock initialized: now 1790639051476031 us; error 49 us; skew 500 ppm
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:11.479259 28406 init.cc:377] Logged in from keytab as oryx/127.21.112.67@KRBTEST.COM (short username oryx)
I20260928 23:44:11.480160 28406 webserver.cc:460] Webserver started at http://127.21.112.67:45599/ using document root <none> and password file <none>
I20260928 23:44:11.480741 28406 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:11.480829 28406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:11.481055 28406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:11.483153 28406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "aa63edaf2320442198a32ec812e069b3"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "e2d846031dbb18799d2df6f968e44cbf"
server_key_iv: "27f30315ae974136f73eee156e885037"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.483719 28406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "aa63edaf2320442198a32ec812e069b3"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "e2d846031dbb18799d2df6f968e44cbf"
server_key_iv: "27f30315ae974136f73eee156e885037"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.487507 28406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:44:11.490020 28421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.491233 28406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:11.491381 28406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "aa63edaf2320442198a32ec812e069b3"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "e2d846031dbb18799d2df6f968e44cbf"
server_key_iv: "27f30315ae974136f73eee156e885037"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.491497 28406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:11.503153 28406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:11.506409 28406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:11.506615 28406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:11.507215 28406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:11.508008 28406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:11.508085 28406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.508157 28406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:11.508208 28406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.517026 28406 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:37599
I20260928 23:44:11.517050 28534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:37599 every 8 connection(s)
I20260928 23:44:11.517935 28406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:11.524950 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28406
I20260928 23:44:11.525089 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:11.525372 21953 external_mini_cluster.cc:1428] Setting key c8f26c2937913253b707dcd342ce6695
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:11.532172 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.519746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:58075 (local address 127.21.112.126:46615)
0928 23:44:11.519924 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.519927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.520846 (+ 919us) server_negotiation.cc:407] Connection header received
0928 23:44:11.522306 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.522309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.522355 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.522446 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.523622 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.524233 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.525767 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.526081 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.526201 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.528756 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.528762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.528770 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.528794 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.530173 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.530691 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.530695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.530697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.530796 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.531148 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.531151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.531153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.531327 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.531433 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.531906 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.532013 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 23:44:11.533011 28537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.520156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:58075)
0928 23:44:11.520688 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.520732 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.522089 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.522604 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.522610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.522949 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.523476 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.523489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.524345 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.524347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.525639 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.525649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.526250 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.526254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.526439 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.527037 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.527053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.528581 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.530297 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.530303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.530318 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.530583 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.530895 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.530898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.530900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.531044 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.531430 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.531438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.531677 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.531867 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.532192 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260928 23:44:11.534085 28535 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46615
I20260928 23:44:11.534305 28535 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:11.534744 28535 heartbeater.cc:511] Master 127.21.112.126:46615 requested a full tablet report, sending...
I20260928 23:44:11.535844 28076 ts_manager.cc:194] Registered new tserver with Master: aa63edaf2320442198a32ec812e069b3 (127.21.112.67:37599)
I20260928 23:44:11.536427 28076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:58075
I20260928 23:44:11.539384 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.112.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.125.keytab.
Entry for principal oryx/127.21.112.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.125.keytab.
I20260928 23:44:11.565986 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:37867
--webserver_interface=127.21.112.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.125.keytab
--principal=oryx/127.21.112.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46615,127.21.112.125:37867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:11.680192 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:11.680497 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:11.680580 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:11.684724 28540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:11.684796 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:11.684816 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:11.684834 28540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:11.684849 28540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:11.691088 28540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46615,127.21.112.125:37867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.125.keytab
--principal=oryx/127.21.112.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:37867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.112.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
W20260928 23:44:11.691335 28540 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.112.126:46615,127.21.112.125:37867'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 23:44:11.692270 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:11.693427 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:11.702122 28546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.702203 28545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.702170 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:11.702469 28540 server_base.cc:1034] running on GCE node
I20260928 23:44:11.703001 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:11.704155 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:11.705353 28540 hybrid_clock.cc:648] HybridClock initialized: now 1790639051705336 us; error 32 us; skew 500 ppm
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:11.708235 28540 init.cc:377] Logged in from keytab as oryx/127.21.112.125@KRBTEST.COM (short username oryx)
I20260928 23:44:11.709110 28540 webserver.cc:460] Webserver started at http://127.21.112.125:34325/ using document root <none> and password file <none>
I20260928 23:44:11.709748 28540 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:11.709841 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:11.710069 28540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:11.712155 28540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/instance:
uuid: "e0fb094665de4669a706ba2fd87a3c69"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "4071ee956bf667964bdf53db200d218e"
server_key_iv: "f0afd58b5ab9e1e990176496a1cf247e"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.712711 28540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal/instance:
uuid: "e0fb094665de4669a706ba2fd87a3c69"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "4071ee956bf667964bdf53db200d218e"
server_key_iv: "f0afd58b5ab9e1e990176496a1cf247e"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.716440 28540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 23:44:11.718878 28555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.720212 28540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:11.720346 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
uuid: "e0fb094665de4669a706ba2fd87a3c69"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "4071ee956bf667964bdf53db200d218e"
server_key_iv: "f0afd58b5ab9e1e990176496a1cf247e"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.720455 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:11.748776 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:11.752236 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:11.752460 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:11.759717 28540 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.125:37867
I20260928 23:44:11.760015 28607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.125:37867 every 8 connection(s)
I20260928 23:44:11.760800 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
I20260928 23:44:11.762763 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28540
I20260928 23:44:11.764014 28608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:11.768514 28608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.125@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:11.780058 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.764377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.21.112.125:37867)
0928 23:44:11.764842 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.764860 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.764898 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:44:11.765639 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.765660 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.766014 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.766361 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.767316 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.768554 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.769775 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.770360 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.770525 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.773988 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.773993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.774003 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.774032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.776088 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.776586 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.776592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.776599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.776721 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.777060 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.777063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.777065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.777605 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.778118 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.778312 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.778926 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":142,"threads_started":1}
I20260928 23:44:11.781229 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.770609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.21.112.126:46615)
0928 23:44:11.770727 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.770732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.771187 (+ 455us) server_negotiation.cc:407] Connection header received
0928 23:44:11.771380 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.771383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.771433 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.771484 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.772260 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.772884 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.774289 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.774576 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.774711 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.777468 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.777472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.777475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.777514 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.779260 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.779789 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.779793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.779795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.779892 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.780195 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.780198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.780200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.780458 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.780530 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.780961 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.781079 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:44:11.781262 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.770575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:38020)
0928 23:44:11.771057 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.771109 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.771276 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.771657 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.771661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.772008 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.772133 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.772138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.773058 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.773062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.774148 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.774159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.774715 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.774719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.774837 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.775510 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.775525 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.777257 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.779381 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.779390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.779395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.779680 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.779990 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.779993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.779995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.780089 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.780608 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.780613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.780726 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.780867 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.781042 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":299,"threads_started":1}
I20260928 23:44:11.782886 28608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.125@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
I20260928 23:44:11.791113 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.783410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:49078)
0928 23:44:11.783513 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.783526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.783597 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.783865 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.783867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.784043 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.784168 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.784173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.785299 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.785302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.785882 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.785890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.786344 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.786346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.786415 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.786860 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.786871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.788192 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.789908 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.789911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.789912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.790096 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.790320 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.790322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.790324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.790391 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.790701 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.790703 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.790775 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.790856 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.790952 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:44:11.791145 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.783446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.21.112.125:37867)
0928 23:44:11.783545 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.783549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.783564 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:11.783718 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.783721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.783758 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.783830 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.784278 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.785191 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.785983 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.786247 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.786330 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.788317 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.788320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.788322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.788341 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.789819 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.790176 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.790178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.790180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.790245 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.790462 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.790463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.790465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.790625 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.790695 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.790908 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.791018 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":36}
I20260928 23:44:11.794512 28608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Bootstrap starting.
I20260928 23:44:11.797098 28608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:11.797909 28608 log.cc:824] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:11.799857 28608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: No bootstrap required, opened a new log
I20260928 23:44:11.802899 28608 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } }
I20260928 23:44:11.803133 28608 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:11.803216 28608 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fb094665de4669a706ba2fd87a3c69, State: Initialized, Role: FOLLOWER
I20260928 23:44:11.803650 28608 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } }
I20260928 23:44:11.804150 28614 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } } }
I20260928 23:44:11.804260 28614 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:11.804832 28608 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:44:11.809299 28625 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:44:11.809379 28625 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:44:11.811331 28540 master_runner.cc:428] Detected that this master e0fb094665de4669a706ba2fd87a3c69 is joining an existing cluster
I20260928 23:44:11.811383 28540 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.112.125:37867
I20260928 23:44:11.821660 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.813435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.21.112.126:46615)
0928 23:44:11.813611 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.813616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.813869 (+ 253us) server_negotiation.cc:407] Connection header received
0928 23:44:11.814007 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.814009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.814050 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.814145 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.814724 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.815515 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.816408 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.816646 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.816757 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.818094 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.818097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.818100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.818132 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.819759 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820294 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.820298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.820300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.820384 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820725 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.820728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.820730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.820954 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.821065 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.821349 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.821472 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:44:11.821659 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.813777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.21.112.125:37867)
0928 23:44:11.813895 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.813899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.814176 (+ 277us) server_negotiation.cc:407] Connection header received
0928 23:44:11.814298 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.814301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.814343 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.814423 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.814951 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.815643 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.816805 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.817120 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.817189 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.818443 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.818448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.818452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.818480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.820023 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820486 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.820489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.820491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.820577 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820838 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.820841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.820843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.821015 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.821092 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.821383 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.821476 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":43}
I20260928 23:44:11.821951 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.813386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:49094)
0928 23:44:11.814110 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.814122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.814224 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.814459 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.814462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.814699 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.814833 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.814840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.815759 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.815763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.816680 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.816689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.817317 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.817321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.817414 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.817893 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.817918 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.818324 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.820130 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.820139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.820381 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.820649 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.820652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.820746 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.821145 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.821149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.821216 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.821677 (+ 461us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.821809 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"spinlock_wait_cycles":1920,"thread_start_us":99,"threads_started":1}
I20260928 23:44:11.821648 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.813365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:38030)
0928 23:44:11.813792 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.813808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.813911 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.814160 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.814163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.814383 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.814598 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.814623 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.815647 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.815650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.816286 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.816294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.816765 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.816768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.816857 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.817393 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.817410 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.817928 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.819883 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.819886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.819888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.820190 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.820479 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.820482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.820484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.820623 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.821051 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.821054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.821159 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.821271 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.821401 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":40,"thread_start_us":95,"threads_started":1}
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.126@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
I20260928 23:44:11.836515 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.826706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.21.112.125:37867)
0928 23:44:11.826822 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.826826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.827434 (+ 608us) server_negotiation.cc:407] Connection header received
0928 23:44:11.827582 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.827585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.827635 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.827703 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.828323 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.829080 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.830044 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.830373 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.830475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.832759 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.832763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.832766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.832790 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.834416 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.834937 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.834942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.834944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.835019 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.835406 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.835412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.835418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.835630 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.835738 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.836234 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.836347 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:44:11.836655 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.826672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:49102)
0928 23:44:11.827325 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.827363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.827471 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.827764 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.827769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.828089 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.828212 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.828218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.829204 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.829207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.829938 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.829948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.830483 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.830485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.830562 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.831089 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.831101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.832628 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.834534 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.834543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.834549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.834832 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.835153 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.835156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.835158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.835241 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.835806 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.835816 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.835998 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.836173 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.836390 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":65,"threads_started":1}
I20260928 23:44:11.838092 28076 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.112.125:37867: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69"
member_type: NON_VOTER
last_known_addr {
host: "127.21.112.125"
port: 37867
}
attrs {
promote: true
}
}
dest_uuid: "f1b0b16080d14035b8611d345c843120"
cas_config_opid_index: -1
I20260928 23:44:11.838785 28076 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } }
I20260928 23:44:11.840889 28636 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e0fb094665de4669a706ba2fd87a3c69 (127.21.112.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } }
I20260928 23:44:11.841352 28582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:11.842602 28636 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.112.125:37867
I20260928 23:44:11.842980 28582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Refusing update from remote peer f1b0b16080d14035b8611d345c843120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 23:44:11.843025 28540 master.cc:537] Master@127.21.112.125:37867 shutting down...
WARNING: no policy specified for oryx/127.21.112.124@KRBTEST.COM; defaulting to no policy
I20260928 23:44:11.843528 28636 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } }
I20260928 23:44:11.843631 28636 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.124@KRBTEST.COM" created.
I20260928 23:44:11.845932 28637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
W20260928 23:44:11.848660 28061 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 23:44:11.849651 28540 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 23:44:11.849641 28061 consensus_peers.cc:487] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 -> Peer e0fb094665de4669a706ba2fd87a3c69 (127.21.112.125:37867): Couldn't send request to peer e0fb094665de4669a706ba2fd87a3c69. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:44:11.849943 28540 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:11.850050 28540 tablet_replica.cc:330] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: stopping tablet replica
I20260928 23:44:11.856989 28540 master.cc:552] Master@127.21.112.125:37867 shutdown complete.
I20260928 23:44:11.857049 28540 master_runner.cc:305] Clearing existing system tablet
I20260928 23:44:11.857286 28540 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:11.857342 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:11.858630 28540 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:11.859730 28642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.859977 28540 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:11.860113 28540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:11.860208 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
uuid: "e0fb094665de4669a706ba2fd87a3c69"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "4071ee956bf667964bdf53db200d218e"
server_key_iv: "f0afd58b5ab9e1e990176496a1cf247e"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.862253 28540 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:11.864730 28540 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:44:11.864853 28540 log.cc:1196] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 23:44:11.865128 28540 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Deleting consensus metadata
I20260928 23:44:11.865347 28540 master_runner.cc:315] Copying system tablet from 127.21.112.126:46615
I20260928 23:44:11.866720 28540 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: tablet copy: Beginning tablet copy session from remote peer at address 127.21.112.126:46615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.124.keytab.
Entry for principal oryx/127.21.112.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.124.keytab.
I20260928 23:44:11.871011 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:46621
--webserver_interface=127.21.112.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.124.keytab
--principal=oryx/127.21.112.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:44:11.876993 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.867512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.21.112.126:46615)
0928 23:44:11.867646 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:11.867652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:11.867793 (+ 141us) server_negotiation.cc:407] Connection header received
0928 23:44:11.867925 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:11.867928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:11.867982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:11.868068 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:11.868615 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.869460 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.870921 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.871342 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.871487 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.873335 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:11.873339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:11.873342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:11.873369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:11.875171 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.875670 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.875674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.875676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.875758 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:11.876045 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:11.876048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:11.876050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:11.876276 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:11.876346 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:11.876702 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 23:44:11.876824 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:11.877794 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:11.867409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:38040)
0928 23:44:11.867682 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:11.867698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:11.867821 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:11.868112 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:11.868116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:11.868366 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:11.868488 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.868501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.869861 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.869865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.870782 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:11.870792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:11.871479 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:11.871483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:11.871592 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:11.872701 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:11.872717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:11.873170 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:11.875296 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.875299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.875302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.875560 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.875862 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:11.875865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:11.875867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:11.875945 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:11.876415 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:11.876418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:11.876527 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:44:11.877389 (+ 862us) client_negotiation.cc:240] Negotiation successful
0928 23:44:11.877595 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":123,"threads_started":1}
I20260928 23:44:11.878326 28096 tablet_copy_service.cc:140] P f1b0b16080d14035b8611d345c843120: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e0fb094665de4669a706ba2fd87a3c69 ({username='oryx', principal='oryx/127.21.112.125@KRBTEST.COM'} at 127.0.0.1:38040)
I20260928 23:44:11.878528 28096 tablet_copy_service.cc:161] P f1b0b16080d14035b8611d345c843120: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e0fb094665de4669a706ba2fd87a3c69 at {username='oryx', principal='oryx/127.21.112.125@KRBTEST.COM'} at 127.0.0.1:38040: session id = e0fb094665de4669a706ba2fd87a3c69-00000000000000000000000000000000
I20260928 23:44:11.880439 28096 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:44:11.882500 28540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:44:11.885264 28540 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: tablet copy: Starting download of 0 data blocks...
I20260928 23:44:11.885499 28540 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: tablet copy: Starting download of 1 WAL segments...
I20260928 23:44:11.887487 28540 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:44:11.889475 28096 tablet_copy_service.cc:342] P f1b0b16080d14035b8611d345c843120: Request end of tablet copy session e0fb094665de4669a706ba2fd87a3c69-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.112.125@KRBTEST.COM'} at 127.0.0.1:38040
I20260928 23:44:11.889662 28096 tablet_copy_service.cc:433] P f1b0b16080d14035b8611d345c843120: ending tablet copy session e0fb094665de4669a706ba2fd87a3c69-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e0fb094665de4669a706ba2fd87a3c69
I20260928 23:44:11.897982 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:11.900385 28650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.900460 28653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:11.900712 28651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:11.901124 28540 server_base.cc:1034] running on GCE node
I20260928 23:44:11.901299 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:11.901666 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:11.902829 28540 hybrid_clock.cc:648] HybridClock initialized: now 1790639051902807 us; error 41 us; skew 500 ppm
Sep 28 23:44:11 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639051, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:11.904481 28540 init.cc:377] Logged in from keytab as oryx/127.21.112.125@KRBTEST.COM (short username oryx)
I20260928 23:44:11.904969 28540 webserver.cc:460] Webserver started at http://127.21.112.125:34203/ using document root <none> and password file <none>
I20260928 23:44:11.905184 28540 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:11.905228 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:11.906767 28540 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:11.907893 28660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:11.908182 28540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:11.908268 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
uuid: "e0fb094665de4669a706ba2fd87a3c69"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "4071ee956bf667964bdf53db200d218e"
server_key_iv: "f0afd58b5ab9e1e990176496a1cf247e"
server_key_version: "encryptionkey@0"
I20260928 23:44:11.908354 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:11.922757 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:11.925032 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:11.930150 28540 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.125:37867
I20260928 23:44:11.930183 28712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.125:37867 every 8 connection(s)
I20260928 23:44:11.930739 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
I20260928 23:44:11.932648 28713 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:44:11.933736 28713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Bootstrap starting.
I20260928 23:44:11.944404 28713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:44:11.944767 28713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Bootstrap complete.
I20260928 23:44:11.945461 28713 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } }
I20260928 23:44:11.945638 28713 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0fb094665de4669a706ba2fd87a3c69, State: Initialized, Role: LEARNER
I20260928 23:44:11.945755 28713 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } }
I20260928 23:44:11.946278 28713 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:11.946133 28716 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } }
I20260928 23:44:11.946565 28716 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:44:11.949657 28727 catalog_manager.cc:1259] Loaded cluster ID: 9a93cd82a70147caaac56fd3a8a59bcf
I20260928 23:44:11.949771 28727 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: loading cluster ID for follower catalog manager: success
I20260928 23:44:11.952075 28727 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: acquiring CA information for follower catalog manager: success
I20260928 23:44:11.953797 28727 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:44:12.018656 28649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:12.019052 28649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:12.019111 28649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:12.024959 28649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:12.025035 28649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:12.025058 28649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:12.025076 28649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:12.025092 28649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:12.031299 28649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.124.keytab
--principal=oryx/127.21.112.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:46621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.112.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:12.032439 28649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:12.033478 28649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:12.042877 28732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.042870 28733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.042877 28735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.043228 28649 server_base.cc:1034] running on GCE node
I20260928 23:44:12.043867 28649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:12.045058 28649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:12.046275 28649 hybrid_clock.cc:648] HybridClock initialized: now 1790639052046218 us; error 54 us; skew 500 ppm
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:12.049791 28649 init.cc:377] Logged in from keytab as oryx/127.21.112.124@KRBTEST.COM (short username oryx)
I20260928 23:44:12.050655 28649 webserver.cc:460] Webserver started at http://127.21.112.124:43347/ using document root <none> and password file <none>
I20260928 23:44:12.051179 28649 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:12.051236 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:12.051401 28649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:12.053517 28649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/instance:
uuid: "059d32c2c2f249d0ae3a59c33a610737"
format_stamp: "Formatted at 2026-09-28 23:44:12 on dist-test-slave-c3s8"
server_key: "8df41533bc1f6413ebfd66c2fd4257e5"
server_key_iv: "81fc4029b52af5e8e5a8d4e1f497374d"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.054106 28649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal/instance:
uuid: "059d32c2c2f249d0ae3a59c33a610737"
format_stamp: "Formatted at 2026-09-28 23:44:12 on dist-test-slave-c3s8"
server_key: "8df41533bc1f6413ebfd66c2fd4257e5"
server_key_iv: "81fc4029b52af5e8e5a8d4e1f497374d"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.057807 28649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:44:12.060196 28742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.061468 28649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:12.061678 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
uuid: "059d32c2c2f249d0ae3a59c33a610737"
format_stamp: "Formatted at 2026-09-28 23:44:12 on dist-test-slave-c3s8"
server_key: "8df41533bc1f6413ebfd66c2fd4257e5"
server_key_iv: "81fc4029b52af5e8e5a8d4e1f497374d"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.061787 28649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:12.073115 28649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:12.076378 28649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:12.076593 28649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:12.083895 28649 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.124:46621
I20260928 23:44:12.084182 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.124:46621 every 8 connection(s)
I20260928 23:44:12.084964 28649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
I20260928 23:44:12.087769 28795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:12.088960 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28649
I20260928 23:44:12.091913 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:44:12.093422 28267 heartbeater.cc:503] Master 127.21.112.126:46615 was elected leader, sending a full tablet report...
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.112.124@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.124@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:12.104260 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.094036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.21.112.126:46615)
0928 23:44:12.094173 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.094177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.094951 (+ 774us) server_negotiation.cc:407] Connection header received
0928 23:44:12.095160 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.095164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.095242 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.095319 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.096008 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.096571 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.097663 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.098073 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.098197 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.100880 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.100883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.100885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.100907 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.102415 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.102939 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.102944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.102945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.103044 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.103339 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.103344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.103345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.103534 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.103643 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.103996 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.104101 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:44:12.104946 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.093872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:38050)
0928 23:44:12.094822 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.094879 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.095034 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.095362 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.095367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.095766 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.095885 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.095891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.096693 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.096697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.097515 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.097523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.098218 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.098221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.098299 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.098950 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.098975 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.100736 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.102572 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.102583 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.102589 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.102834 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.103152 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.103155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.103157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.103243 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.103642 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.103649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.103759 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.103924 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.104088 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":42,"thread_start_us":158,"threads_started":1}
I20260928 23:44:12.104946 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.090466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.21.112.124:46621)
0928 23:44:12.090889 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.090907 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.090950 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:44:12.091623 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.091640 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.091998 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.092330 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.093307 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.095145 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.096181 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.096693 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.096839 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.099852 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.099859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.099873 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.099902 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.101964 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.102402 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.102408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.102415 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.102530 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.102808 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.102812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.102814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.103225 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.103449 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.103694 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.103985 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":281,"thread_start_us":135,"threads_started":1}
I20260928 23:44:12.106334 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.124@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
I20260928 23:44:12.114347 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.106789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:49110)
0928 23:44:12.106898 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.106914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.106995 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.107416 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.107418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.107546 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.107627 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.107632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.108660 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.108668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.109201 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.109208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.109766 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.109770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.109867 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.110240 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.110250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.111383 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.113141 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.113144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.113145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.113315 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.113613 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.113615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.113616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.113685 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.113991 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.113993 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.114070 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.114149 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.114239 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 23:44:12.114403 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.106923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.21.112.125:37867)
0928 23:44:12.107219 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.107223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.107234 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:44:12.107269 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.107271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.107309 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.107387 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.107770 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.108562 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.109325 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.109650 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.109768 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.111508 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.111511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.111513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.111535 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.113045 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.113393 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.113396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.113397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.113476 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.113758 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.113760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.113761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.113916 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.113992 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.114199 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.114289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260928 23:44:12.115214 28795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.124@KRBTEST.COM for oryx/127.21.112.124@KRBTEST.COM
I20260928 23:44:12.123420 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.115891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.21.112.124:46621)
0928 23:44:12.116015 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.116019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.116033 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:12.116236 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.116239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.116279 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.116356 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.116688 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.117505 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.118250 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.118474 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.118589 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.120531 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.120540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.120542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.120560 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.121941 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.122348 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.122354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.122356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.122422 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.122680 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.122683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.122685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.122837 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.122934 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.123172 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.123304 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":55}
I20260928 23:44:12.123420 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.115705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:35242)
0928 23:44:12.115956 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.115971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.116112 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.116363 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.116366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.116520 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.116584 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.116588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.117644 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.117650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.118140 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.118147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.118588 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.118591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.118661 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.119126 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.119136 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.120389 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.122075 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.122078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.122080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.122254 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.122505 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.122509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.122512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.122591 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.122934 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.122936 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.123024 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.123121 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.123250 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260928 23:44:12.126412 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Bootstrap starting.
I20260928 23:44:12.128633 28795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:12.129311 28795 log.cc:824] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:12.131261 28795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: No bootstrap required, opened a new log
I20260928 23:44:12.134222 28795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } }
I20260928 23:44:12.134420 28795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:12.134454 28795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059d32c2c2f249d0ae3a59c33a610737, State: Initialized, Role: FOLLOWER
I20260928 23:44:12.134866 28795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } }
I20260928 23:44:12.135342 28801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } } }
I20260928 23:44:12.135484 28801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:12.136065 28795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:44:12.141044 28812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:44:12.141155 28812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:44:12.145403 28649 master_runner.cc:428] Detected that this master 059d32c2c2f249d0ae3a59c33a610737 is joining an existing cluster
I20260928 23:44:12.145478 28649 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.112.124:46621
I20260928 23:44:12.157933 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.147961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:49120)
0928 23:44:12.148419 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.148434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.148543 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.149209 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.149212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.149522 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.149683 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.149692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.150793 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.150796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.152230 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.152242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.152978 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152991 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.153092 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.153738 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.153752 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.154251 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.156214 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.156218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.156220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.156473 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.156793 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.156796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.156798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.156888 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.157340 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.157344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.157448 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.157612 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.157776 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":119,"thread_start_us":271,"threads_started":1}
I20260928 23:44:12.157974 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.148040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.21.112.125:37867)
0928 23:44:12.148164 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.148168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.148524 (+ 356us) server_negotiation.cc:407] Connection header received
0928 23:44:12.148941 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.148944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.148993 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.149060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.149820 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.150646 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152366 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.152713 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152789 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.154397 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.154400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.154402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.154427 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.156103 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.156597 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.156600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.156601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.156696 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.156999 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.157002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.157003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.157202 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.157339 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.157675 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.157809 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:44:12.158953 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28138
I20260928 23:44:12.164422 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:46149
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:46615
--webserver_port=37437
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:44:12.179351 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.148616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:35256)
0928 23:44:12.148969 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.148983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.149163 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.149518 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.149521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.149742 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.149847 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.149855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.150956 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.150959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.152324 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.152335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.153068 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.153074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.153176 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.174380 (+ 21204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.174399 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.174995 (+ 596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.177271 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.177273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.177275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.177740 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.178101 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.178103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.178105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.178295 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.178686 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.178689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.178854 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.179091 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.179184 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":257,"spinlock_wait_cycles":7936,"thread_start_us":85,"threads_started":1}
I20260928 23:44:12.179580 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.148616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.21.112.124:46621)
0928 23:44:12.148739 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.148743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.149042 (+ 299us) server_negotiation.cc:407] Connection header received
0928 23:44:12.149262 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.149265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.149309 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.149363 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.149954 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.150817 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152457 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.152798 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152871 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.175173 (+ 22302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.175178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.175181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.175205 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.177155 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.177891 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.177894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.177896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.177994 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.178380 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.178383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.178385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.178561 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.178644 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.179037 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.179423 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":285,"server-negotiator.queue_time_us":45}
I20260928 23:44:12.180567 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.147976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.21.112.126:46615)
0928 23:44:12.148116 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.148121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.148796 (+ 675us) server_negotiation.cc:407] Connection header received
0928 23:44:12.148899 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.148903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.148964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.149062 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.149924 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.150821 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152470 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.152856 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.152928 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.175143 (+ 22215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.175147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.175151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.175185 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.177111 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.177874 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.177879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.177881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.177977 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.178297 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.178301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.178303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.178562 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.178648 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.180292 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.180414 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:44:12.179355 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.148116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:38056)
0928 23:44:12.148716 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.148728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.148813 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.149191 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.149197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.149625 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.149766 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.149774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.150956 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.150959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.152324 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.152335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.153096 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.153099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.153217 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.174261 (+ 21044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.174288 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.174890 (+ 602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.177249 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.177254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.177257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.177754 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.178086 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.178090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.178093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.178192 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.178687 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.178689 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.178854 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.178967 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.179131 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":195,"spinlock_wait_cycles":10496,"thread_start_us":112,"threads_started":1}
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639050, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.126@KRBTEST.COM for oryx/127.21.112.124@KRBTEST.COM
I20260928 23:44:12.193043 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.185177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.21.112.124:46621)
0928 23:44:12.185284 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.185289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.185365 (+ 76us) server_negotiation.cc:407] Connection header received
0928 23:44:12.185583 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.185586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.185660 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.185746 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.186154 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.186667 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.187450 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.187687 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.187796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.189918 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.189922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.189924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.189948 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.191355 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.191793 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.191799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.191801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.191879 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.192177 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.192182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.192184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.192359 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.192465 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.192730 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.192875 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:44:12.193044 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.185139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:35260)
0928 23:44:12.185309 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.185322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.185409 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.185763 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.185766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.185954 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.186038 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.186044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.186772 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.186777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.187335 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.187343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.187798 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.187802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.187895 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.188451 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.188462 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.189786 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.191483 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.191486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.191488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.191697 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.191977 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.191980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.191982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.192069 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.192455 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.192458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.192561 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.192670 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.192818 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 23:44:12.194515 28076 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.112.124:46621: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737"
member_type: NON_VOTER
last_known_addr {
host: "127.21.112.124"
port: 46621
}
attrs {
promote: true
}
}
dest_uuid: "f1b0b16080d14035b8611d345c843120"
cas_config_opid_index: 5
I20260928 23:44:12.195178 28076 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } }
I20260928 23:44:12.196381 28637 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.197265 28769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:12.197954 28637 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.112.124:46621
I20260928 23:44:12.198118 28637 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.198206 28637 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.198750 28769 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Refusing update from remote peer f1b0b16080d14035b8611d345c843120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:44:12.198764 28649 master.cc:537] Master@127.21.112.124:46621 shutting down...
I20260928 23:44:12.199507 28636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:44:12.202083 28061 consensus_peers.cc:487] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 -> Peer 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124:46621): Couldn't send request to peer 059d32c2c2f249d0ae3a59c33a610737. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:44:12.206135 28649 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:12.206427 28649 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:12.206548 28649 tablet_replica.cc:330] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: stopping tablet replica
I20260928 23:44:12.206770 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.196695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.21.112.125:37867)
0928 23:44:12.201249 (+ 4554us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.201256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.201273 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:12.201332 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.201338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.201409 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.201509 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:12.202551 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.203234 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.205027 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.205694 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.205800 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.205940 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.206585 (+ 645us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.206656 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4446}
I20260928 23:44:12.207259 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.196465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:49128)
0928 23:44:12.198854 (+ 2389us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.198869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.198943 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.202294 (+ 3351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.202298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.202320 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:12.202417 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.202426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.203354 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.203357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.204769 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.204784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.206043 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.206048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.206154 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.206291 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.206366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.206450 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2311}
I20260928 23:44:12.207899 28687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 LEARNER]: Refusing update from remote peer f1b0b16080d14035b8611d345c843120: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:44:12.208546 28636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 23:44:12.210767 28687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.212808 28716 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b0b16080d14035b8611d345c843120. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.212893 28716 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:44:12.212976 28716 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: NON_VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: true } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.213018 28716 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:44:12.213977 28636 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: attempting to promote NON_VOTER e0fb094665de4669a706ba2fd87a3c69 to VOTER
I20260928 23:44:12.214416 28636 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } }
I20260928 23:44:12.215802 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.215516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:35262)
0928 23:44:12.215633 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.215726 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 23:44:12.216390 28687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 LEARNER]: Refusing update from remote peer f1b0b16080d14035b8611d345c843120: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 23:44:12.216614 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.216362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:35276)
0928 23:44:12.216497 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.216549 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 23:44:12.216729 28061 consensus_peers.cc:487] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 -> Peer 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124:46621): Couldn't send request to peer 059d32c2c2f249d0ae3a59c33a610737. Status: Network error: Client connection negotiation failed: client connection to 127.21.112.124:46621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:44:12.217077 28824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 23:44:12.218567 28637 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e0fb094665de4669a706ba2fd87a3c69 (127.21.112.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.219779 28637 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.219885 28637 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.220021 28637 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.220127 28637 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.220580 28687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e0fb094665de4669a706ba2fd87a3c69 (127.21.112.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.222342 28823 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.222482 28823 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:12.223186 28649 master.cc:552] Master@127.21.112.124:46621 shutdown complete.
I20260928 23:44:12.223251 28649 master_runner.cc:305] Clearing existing system tablet
I20260928 23:44:12.223479 28649 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:12.223547 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:12.224694 28649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:12.225579 28826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.225838 28649 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:12.225988 28649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:12.226025 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
uuid: "059d32c2c2f249d0ae3a59c33a610737"
format_stamp: "Formatted at 2026-09-28 23:44:12 on dist-test-slave-c3s8"
server_key: "8df41533bc1f6413ebfd66c2fd4257e5"
server_key_iv: "81fc4029b52af5e8e5a8d4e1f497374d"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.227644 28649 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:12.229455 28649 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:44:12.229596 28649 log.cc:1196] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 23:44:12.229842 28649 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Deleting consensus metadata
I20260928 23:44:12.230021 28649 master_runner.cc:315] Copying system tablet from 127.21.112.126:46615
I20260928 23:44:12.231118 28649 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: tablet copy: Beginning tablet copy session from remote peer at address 127.21.112.126:46615
I20260928 23:44:12.239287 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.231721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.21.112.126:46615)
0928 23:44:12.231934 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.231938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.232134 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:44:12.232279 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.232282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.232355 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.232433 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.232870 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.233421 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.234284 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.234515 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.234621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.235759 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.235763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.235766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.235790 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.237409 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.237941 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.237944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.237946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.238031 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.238363 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.238366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.238368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.238581 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.238665 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.239012 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.239136 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260928 23:44:12.239298 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.231659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:38070)
0928 23:44:12.232038 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.232049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.232181 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.232448 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.232451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.232659 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.232751 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.232756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.233530 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.233561 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.234159 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.234168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.234632 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.234636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.234727 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.235267 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.235288 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.235621 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.237513 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.237518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.237521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.237837 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.238136 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.238139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.238141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.238249 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.238753 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.238755 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.238837 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.238941 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.239090 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":84,"threads_started":1}
I20260928 23:44:12.240083 28096 tablet_copy_service.cc:140] P f1b0b16080d14035b8611d345c843120: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 059d32c2c2f249d0ae3a59c33a610737 ({username='oryx', principal='oryx/127.21.112.124@KRBTEST.COM'} at 127.0.0.1:38070)
I20260928 23:44:12.240206 28096 tablet_copy_service.cc:161] P f1b0b16080d14035b8611d345c843120: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 059d32c2c2f249d0ae3a59c33a610737 at {username='oryx', principal='oryx/127.21.112.124@KRBTEST.COM'} at 127.0.0.1:38070: session id = 059d32c2c2f249d0ae3a59c33a610737-00000000000000000000000000000000
I20260928 23:44:12.241039 28096 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:44:12.242231 28649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:44:12.244266 28649 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: tablet copy: Starting download of 0 data blocks...
I20260928 23:44:12.244467 28649 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: tablet copy: Starting download of 1 WAL segments...
I20260928 23:44:12.246359 28649 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:44:12.248040 28096 tablet_copy_service.cc:342] P f1b0b16080d14035b8611d345c843120: Request end of tablet copy session 059d32c2c2f249d0ae3a59c33a610737-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.112.124@KRBTEST.COM'} at 127.0.0.1:38070
I20260928 23:44:12.248140 28096 tablet_copy_service.cc:433] P f1b0b16080d14035b8611d345c843120: ending tablet copy session 059d32c2c2f249d0ae3a59c33a610737-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 059d32c2c2f249d0ae3a59c33a610737
I20260928 23:44:12.254468 28649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:12.256150 28833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.256259 28834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.256443 28649 server_base.cc:1034] running on GCE node
W20260928 23:44:12.256469 28836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.256712 28649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:12.257063 28649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:12.258198 28649 hybrid_clock.cc:648] HybridClock initialized: now 1790639052258189 us; error 28 us; skew 500 ppm
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:12.259629 28649 init.cc:377] Logged in from keytab as oryx/127.21.112.124@KRBTEST.COM (short username oryx)
I20260928 23:44:12.260144 28649 webserver.cc:460] Webserver started at http://127.21.112.124:35041/ using document root <none> and password file <none>
I20260928 23:44:12.260329 28649 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:12.260382 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:12.261476 28649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:12.262382 28843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.262615 28649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:12.262706 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
uuid: "059d32c2c2f249d0ae3a59c33a610737"
format_stamp: "Formatted at 2026-09-28 23:44:12 on dist-test-slave-c3s8"
server_key: "8df41533bc1f6413ebfd66c2fd4257e5"
server_key_iv: "81fc4029b52af5e8e5a8d4e1f497374d"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.262814 28649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:12.291589 28649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:12.294006 28649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:12.302701 28649 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.124:46621
I20260928 23:44:12.303256 28649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
I20260928 23:44:12.305979 28895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.124:46621 every 8 connection(s)
I20260928 23:44:12.307261 28896 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:44:12.308180 28896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Bootstrap starting.
I20260928 23:44:12.317413 28896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:44:12.317754 28896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: Bootstrap complete.
I20260928 23:44:12.318280 28896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } }
I20260928 23:44:12.318414 28896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 059d32c2c2f249d0ae3a59c33a610737, State: Initialized, Role: LEARNER
I20260928 23:44:12.318529 28896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } }
I20260928 23:44:12.318907 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.319103 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:44:12.319105 28896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:12.322798 28913 catalog_manager.cc:1259] Loaded cluster ID: 9a93cd82a70147caaac56fd3a8a59bcf
I20260928 23:44:12.322845 28913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: loading cluster ID for follower catalog manager: success
I20260928 23:44:12.323612 28401 heartbeater.cc:503] Master 127.21.112.126:46615 was elected leader, sending a full tablet report...
I20260928 23:44:12.325146 28913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: acquiring CA information for follower catalog manager: success
I20260928 23:44:12.326354 28913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:44:12.326212 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:12.326540 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:12.326613 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:12.332669 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:12.332744 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:12.332832 28821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:12.341109 28821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:46149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=37437
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:12.342455 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:12.343699 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:44:12.355284 28821 server_base.cc:1034] running on GCE node
W20260928 23:44:12.355223 28915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.355237 28918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.355206 28916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.355795 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:12.356447 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:12.357651 28821 hybrid_clock.cc:648] HybridClock initialized: now 1790639052357617 us; error 48 us; skew 500 ppm
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:12.360441 28821 init.cc:377] Logged in from keytab as oryx/127.21.112.65@KRBTEST.COM (short username oryx)
I20260928 23:44:12.361383 28821 webserver.cc:460] Webserver started at http://127.21.112.65:37437/ using document root <none> and password file <none>
I20260928 23:44:12.362057 28821 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:12.362165 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:12.367066 28821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:12.369226 28925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.370520 28821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:12.370677 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "f87a2836595e4202906b910e99da3dd1"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c6d27707d28853b383f5c3cd03cecf59"
server_key_iv: "ebaf371a33e418645cf4b21cbc18de29"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.371246 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:12.424934 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:12.428419 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:12.428646 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:12.429271 28821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:12.430184 28821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:12.430258 28821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.430330 28821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:12.430374 28821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.439895 28821 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:46149
I20260928 23:44:12.439939 29038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:46149 every 8 connection(s)
I20260928 23:44:12.440961 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:12.447077 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 28821
I20260928 23:44:12.447321 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28272
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.124@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:12.457657 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:46087
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:46615
--webserver_port=38731
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:44:12.459769 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.444600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:48843 (local address 127.21.112.125:37867)
0928 23:44:12.444767 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.444771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.444785 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:12.445570 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.445576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.445626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.445711 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.447309 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.448134 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450375 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.450714 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450791 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.454663 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.454670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.454676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.454709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.457036 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.457780 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.457786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.457790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.457879 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.458266 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.458272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.458277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.458528 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.458615 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.459425 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.459566 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:44:12.459771 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.443043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:44677 (local address 127.21.112.124:46621)
0928 23:44:12.443325 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.443329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.444026 (+ 697us) server_negotiation.cc:407] Connection header received
0928 23:44:12.445289 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.445292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.445343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.445406 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.447298 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.448493 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450375 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.450837 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450931 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.454921 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.454925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.454927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.454956 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.456906 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.457726 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.457731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.457733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.457835 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.458278 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.458283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.458285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.458542 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.458637 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.459421 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.459567 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
I20260928 23:44:12.461565 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.442957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:38639 (local address 127.21.112.126:46615)
0928 23:44:12.443109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.443114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.444029 (+ 915us) server_negotiation.cc:407] Connection header received
0928 23:44:12.445267 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.445271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.445326 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.445411 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.447338 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.448134 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450375 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.450714 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450791 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.457338 (+ 6547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.457342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.457344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.457376 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.459395 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.460264 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.460268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.460269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.460360 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.460670 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.460673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.460675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.460882 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.460964 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.461278 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.461392 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 23:44:12.461638 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.444450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.21.112.65:48843)
0928 23:44:12.444698 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.444712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.445057 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.445727 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.445730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.446287 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.447122 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.447138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.448323 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.448327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.450236 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.450247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.450993 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.451338 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.452210 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.452237 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.454463 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.457334 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.457337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.457339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.457660 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.458005 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.458011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.458016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.458157 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.458687 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.458697 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.459108 (+ 411us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.460012 (+ 904us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.460561 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":583,"spinlock_wait_cycles":55168,"thread_start_us":88,"threads_started":1}
I20260928 23:44:12.461635 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.443417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.21.112.65:44677)
0928 23:44:12.443875 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.443928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.445060 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.445649 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.445658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.446288 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.447116 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.447139 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.448669 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.448675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.450235 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.450249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.451124 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.451130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.451345 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.452263 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.452278 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.454750 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.457152 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.457159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.457176 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.457611 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.457950 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.457953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.457955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.458161 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.458709 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.458718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.459105 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.460024 (+ 919us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.460467 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":444,"spinlock_wait_cycles":24448,"thread_start_us":132,"threads_started":1}
I20260928 23:44:12.463092 29041 heartbeater.cc:347] Connected to a master server at 127.21.112.125:37867
I20260928 23:44:12.463151 29039 heartbeater.cc:347] Connected to a master server at 127.21.112.124:46621
I20260928 23:44:12.463357 29039 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.463356 29041 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.463581 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.443417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:38639)
0928 23:44:12.443875 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.443928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.445057 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.445648 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.445654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.446287 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.447121 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.447139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.448325 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.448327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.450236 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.450247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.450993 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.450997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.451315 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.452210 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.452242 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.457172 (+ 4930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.459533 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.459537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.459539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.459767 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.460477 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.460479 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.460481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.460571 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.461018 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.461021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.461101 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.462921 (+ 1820us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.463079 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":670,"spinlock_wait_cycles":125312,"thread_start_us":151,"threads_started":1}
I20260928 23:44:12.463989 29041 heartbeater.cc:511] Master 127.21.112.125:37867 requested a full tablet report, sending...
I20260928 23:44:12.464223 29040 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46615
I20260928 23:44:12.464304 29040 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.464483 29040 heartbeater.cc:511] Master 127.21.112.126:46615 requested a full tablet report, sending...
I20260928 23:44:12.463996 29039 heartbeater.cc:511] Master 127.21.112.124:46621 requested a full tablet report, sending...
I20260928 23:44:12.465380 28076 ts_manager.cc:194] Re-registered known tserver with Master: f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149)
I20260928 23:44:12.465479 28677 ts_manager.cc:194] Registered new tserver with Master: f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149)
I20260928 23:44:12.466320 28076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:38639
I20260928 23:44:12.467320 28860 ts_manager.cc:194] Registered new tserver with Master: f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149)
I20260928 23:44:12.538623 28535 heartbeater.cc:503] Master 127.21.112.126:46615 was elected leader, sending a full tablet report...
W20260928 23:44:12.575240 29047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:12.575555 29047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:12.575600 29047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:12.579679 29047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:12.579753 29047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:12.579815 29047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:12.585325 29047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=38731
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:12.586508 29047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:12.587508 29047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:12.597954 29055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.597955 29053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.598198 29052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.598385 29047 server_base.cc:1034] running on GCE node
I20260928 23:44:12.598809 29047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:12.599473 29047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:12.600643 29047 hybrid_clock.cc:648] HybridClock initialized: now 1790639052600624 us; error 32 us; skew 500 ppm
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:12.603575 29047 init.cc:377] Logged in from keytab as oryx/127.21.112.66@KRBTEST.COM (short username oryx)
I20260928 23:44:12.604509 29047 webserver.cc:460] Webserver started at http://127.21.112.66:38731/ using document root <none> and password file <none>
I20260928 23:44:12.605041 29047 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:12.605113 29047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:12.609112 29047 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:12.611240 29062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.612500 29047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:12.612638 29047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "68d2645e4057458692cde93aaed70e46"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "c4ee9fbc25ac16a091fcd1661006724b"
server_key_iv: "a0e8ed14a291b332bd328b165cb220a3"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.613195 29047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:12.635592 29047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:12.639014 29047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:12.639217 29047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:12.639837 29047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:12.640703 29047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:12.640772 29047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.640844 29047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:12.640897 29047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.650647 29047 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:46087
I20260928 23:44:12.650748 29175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:46087 every 8 connection(s)
I20260928 23:44:12.651743 29047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:12.655112 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 29047
I20260928 23:44:12.655296 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28406
I20260928 23:44:12.664084 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:37599
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:46615
--webserver_port=45599
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.124@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
I20260928 23:44:12.670425 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.653756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:39261 (local address 127.21.112.126:46615)
0928 23:44:12.653915 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.653920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.655826 (+ 1906us) server_negotiation.cc:407] Connection header received
0928 23:44:12.656929 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.656932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.656988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.657070 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.658180 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.659005 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.660992 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.661328 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661418 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.665888 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.665892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.665895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.665927 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.668081 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.668816 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.668821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.668824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.668920 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.669251 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.669255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.669257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.669456 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.669580 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.670142 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.670265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:44:12.670557 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.655914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:60681 (local address 127.21.112.125:37867)
0928 23:44:12.656063 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.656068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.657119 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 23:44:12.657301 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.657307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.657366 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.657422 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.658656 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.659544 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661002 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.661434 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661554 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.666626 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.666629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.666631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.666656 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.668131 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.668737 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.668741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.668743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.668829 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.669176 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.669179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.669181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.669410 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.669518 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.670188 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.670329 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 23:44:12.671420 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.654279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:39261)
0928 23:44:12.655735 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.655751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.656789 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.657193 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.657195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.657375 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.658009 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.658030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.659171 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.659174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.660856 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.660866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.661623 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.661983 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.662962 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.662976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.665706 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.668228 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.668236 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.668260 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.668682 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.669034 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.669039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.669057 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.669158 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.669651 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.669654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.669896 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.670094 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.670439 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":330,"spinlock_wait_cycles":14464,"thread_start_us":151,"threads_started":1}
I20260928 23:44:12.671638 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.655378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.21.112.66:60681)
0928 23:44:12.657027 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.657042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.657188 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.657609 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.657614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.657828 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.658497 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.658510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.659695 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.659701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.660863 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.660876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.661730 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.661940 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.662809 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.662835 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.666451 (+ 3616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.668298 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.668301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.668303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.668633 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.668927 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.668930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.668932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.669073 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.669563 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.669572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.669908 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.671220 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.671399 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"mutex_wait_us":586,"spinlock_wait_cycles":13568,"thread_start_us":79,"threads_started":1}
I20260928 23:44:12.672969 29178 heartbeater.cc:347] Connected to a master server at 127.21.112.125:37867
I20260928 23:44:12.673233 29178 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.673317 29177 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46615
I20260928 23:44:12.673390 29177 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.673897 29178 heartbeater.cc:511] Master 127.21.112.125:37867 requested a full tablet report, sending...
I20260928 23:44:12.673897 29177 heartbeater.cc:511] Master 127.21.112.126:46615 requested a full tablet report, sending...
I20260928 23:44:12.675077 28076 ts_manager.cc:194] Re-registered known tserver with Master: 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
I20260928 23:44:12.675237 28677 ts_manager.cc:194] Registered new tserver with Master: 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
I20260928 23:44:12.675856 28076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:39261
I20260928 23:44:12.676097 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.654113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:49267 (local address 127.21.112.124:46621)
0928 23:44:12.654245 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.654249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.654946 (+ 697us) server_negotiation.cc:407] Connection header received
0928 23:44:12.656338 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.656341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.656400 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.656479 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.658210 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.659005 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.660991 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.661344 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661417 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.670815 (+ 9398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.670822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.670826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.670868 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.672731 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.674785 (+ 2054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.674789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.674791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.674886 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.675215 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.675218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.675220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.675427 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.675510 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.675818 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.675950 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:44:12.676843 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.654279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.21.112.66:49267)
0928 23:44:12.654777 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.654851 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.656108 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.656653 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.656659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.657131 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.658070 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.658081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.659171 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.659174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.660861 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.660866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.661623 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.661627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.661973 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.662851 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.662867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.666134 (+ 3267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.672864 (+ 6730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.672868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.672869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.674665 (+ 1796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.675002 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.675005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.675007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.675106 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.675570 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.675573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.675654 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.676548 (+ 894us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.676683 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":409,"spinlock_wait_cycles":3840,"thread_start_us":592,"threads_started":1}
I20260928 23:44:12.678292 29176 heartbeater.cc:347] Connected to a master server at 127.21.112.124:46621
I20260928 23:44:12.678387 29176 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.678566 29176 heartbeater.cc:511] Master 127.21.112.124:46621 requested a full tablet report, sending...
I20260928 23:44:12.679323 28860 ts_manager.cc:194] Registered new tserver with Master: 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
W20260928 23:44:12.782951 29183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:12.783303 29183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:12.783398 29183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:12.787547 29183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:12.787642 29183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:12.787737 29183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:12.793267 29183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:37599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=45599
--tserver_master_addrs=127.21.112.126:46615,127.21.112.125:37867,127.21.112.124:46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:12.795035 29183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:12.796072 29183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:12.805936 29192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.805923 29183 server_base.cc:1034] running on GCE node
W20260928 23:44:12.805909 29189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:12.805908 29190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:12.806617 29183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:12.807248 29183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:12.808419 29183 hybrid_clock.cc:648] HybridClock initialized: now 1790639052808369 us; error 61 us; skew 500 ppm
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:12.811399 29183 init.cc:377] Logged in from keytab as oryx/127.21.112.67@KRBTEST.COM (short username oryx)
I20260928 23:44:12.812390 29183 webserver.cc:460] Webserver started at http://127.21.112.67:45599/ using document root <none> and password file <none>
I20260928 23:44:12.812966 29183 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:12.813149 29183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:12.817217 29183 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:12.819669 29199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.821338 29183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:44:12.821468 29183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "aa63edaf2320442198a32ec812e069b3"
format_stamp: "Formatted at 2026-09-28 23:44:11 on dist-test-slave-c3s8"
server_key: "e2d846031dbb18799d2df6f968e44cbf"
server_key_iv: "27f30315ae974136f73eee156e885037"
server_key_version: "encryptionkey@0"
I20260928 23:44:12.822124 29183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:12.825017 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.821408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.21.112.124:46621)
0928 23:44:12.821554 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.821560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.821765 (+ 205us) server_negotiation.cc:407] Connection header received
0928 23:44:12.821933 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.821936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.821997 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.822098 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:12.822399 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.822958 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.824000 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.824451 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.824593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.824732 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.824818 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.824885 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:44:12.825039 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.821331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:35284)
0928 23:44:12.821678 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.821691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.821827 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.822124 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.822127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.822146 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:12.822263 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.822270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.823090 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.823092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.823875 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.823884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.824593 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.824597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.824687 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.824713 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.824804 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.824891 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":65,"threads_started":1}
I20260928 23:44:12.825691 28870 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 23:44:12.826151 29201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b0b16080d14035b8611d345c843120. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.826319 29201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:44:12.851532 29183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:12.854807 29183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:12.854960 29183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:12.855504 29183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:12.856312 29183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:12.856359 29183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.856396 29183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:12.856412 29183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:12.865329 29183 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:37599
I20260928 23:44:12.865408 29314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:37599 every 8 connection(s)
I20260928 23:44:12.866524 29183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:12.871572 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 29183
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.124@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.125@KRBTEST.COM
Sep 28 23:44:12 dist-test-slave-c3s8 krb5kdc[28031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639052, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:12.885701 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.868387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:47493 (local address 127.21.112.125:37867)
0928 23:44:12.868560 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.868565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.869475 (+ 910us) server_negotiation.cc:407] Connection header received
0928 23:44:12.870838 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.870841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.870893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.870946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.872829 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.875035 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.876341 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.876969 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.877082 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.880885 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.880888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.880890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.880909 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.882879 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.883519 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.883522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.883524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.883622 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.884002 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.884006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.884007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.884226 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.884312 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.885334 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.885449 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:44:12.886646 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.869901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.21.112.67:35457)
0928 23:44:12.870175 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.870197 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.870623 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.871180 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.871188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.871833 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.872650 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.872670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.873775 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.873779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.875218 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.875225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.875950 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.875953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.876247 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.876882 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.876904 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.881143 (+ 4239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.883657 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.883661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.883664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.883926 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.884270 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.884275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.884279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.884373 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.884824 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.884826 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.884908 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.885000 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.885353 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":157,"spinlock_wait_cycles":30848,"thread_start_us":100,"threads_started":1}
I20260928 23:44:12.887360 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.870058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:35457 (local address 127.21.112.124:46621)
0928 23:44:12.870177 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.870183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.870283 (+ 100us) server_negotiation.cc:407] Connection header received
0928 23:44:12.870838 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.870841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.870894 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.870946 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.872823 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.873610 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.875342 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.875691 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.875791 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.881262 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.881269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.881275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.881304 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.883551 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.884053 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.884058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.884061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.884163 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.884482 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.884487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.884491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.884708 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.884785 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.887080 (+ 2295us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.887204 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:44:12.888120 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.868859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.21.112.67:47493)
0928 23:44:12.869325 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.869377 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.870623 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.871180 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.871188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.871833 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.872646 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.872668 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.875242 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.875245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.876214 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.876224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.877101 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.877104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.877265 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.878537 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.878554 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.880727 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.883014 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.883021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.883039 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.883395 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.883696 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.883700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.883704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.883890 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.884363 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.884370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.884706 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.884938 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.887572 (+ 2634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":2457,"spinlock_wait_cycles":30592,"thread_start_us":102,"threads_started":1}
I20260928 23:44:12.888652 29315 heartbeater.cc:347] Connected to a master server at 127.21.112.124:46621
I20260928 23:44:12.888691 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.874523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.21.112.125:37867)
0928 23:44:12.874764 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.874768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.874781 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:44:12.874816 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.874820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.874871 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.875934 (+ 1063us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.876023 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.876740 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.878760 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.879154 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.879257 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.880837 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.880843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.880848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.880877 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.886259 (+ 5382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.886697 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.886701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.886702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.886796 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.887397 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.887403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.887407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.887636 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.887760 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.888366 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.888483 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260928 23:44:12.888901 29315 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.889027 29317 heartbeater.cc:347] Connected to a master server at 127.21.112.125:37867
I20260928 23:44:12.889110 29317 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.889287 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.868576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:57721 (local address 127.21.112.126:46615)
0928 23:44:12.868708 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.868712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.869486 (+ 774us) server_negotiation.cc:407] Connection header received
0928 23:44:12.870823 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.870826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.870885 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.870964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.874202 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.875019 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.877443 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.878090 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.878168 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.882604 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.882609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.882614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.882646 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.884577 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.887763 (+ 3186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.887767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.887769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.887868 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.888332 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.888337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.888341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.888574 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.888672 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.889009 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.889149 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:44:12.889511 29315 heartbeater.cc:511] Master 127.21.112.124:46621 requested a full tablet report, sending...
I20260928 23:44:12.889645 29317 heartbeater.cc:511] Master 127.21.112.125:37867 requested a full tablet report, sending...
I20260928 23:44:12.889815 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.868859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:57721)
0928 23:44:12.869326 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.869379 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.870622 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.871180 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.871188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.871854 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:12.874005 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.874072 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.875221 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.875224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.877308 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.877317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.878333 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.878336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.878534 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.879003 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:12.879017 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:12.882420 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:12.886947 (+ 4527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.886950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.886952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.887181 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.888147 (+ 966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:12.888150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:12.888152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:12.888240 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:12.888716 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:12.888719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:12.888815 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.889418 (+ 603us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.889617 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":2708,"spinlock_wait_cycles":40832,"thread_start_us":150,"threads_started":1}
I20260928 23:44:12.890453 29316 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46615
I20260928 23:44:12.890523 29316 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:12.890666 29316 heartbeater.cc:511] Master 127.21.112.126:46615 requested a full tablet report, sending...
I20260928 23:44:12.891179 28677 ts_manager.cc:194] Registered new tserver with Master: aa63edaf2320442198a32ec812e069b3 (127.21.112.67:37599)
I20260928 23:44:12.891237 28860 ts_manager.cc:194] Registered new tserver with Master: aa63edaf2320442198a32ec812e069b3 (127.21.112.67:37599)
I20260928 23:44:12.891700 28076 ts_manager.cc:194] Re-registered known tserver with Master: aa63edaf2320442198a32ec812e069b3 (127.21.112.67:37599)
I20260928 23:44:12.892481 28076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:57721
I20260928 23:44:12.893760 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.873837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.21.112.126:46615)
0928 23:44:12.877760 (+ 3923us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.877766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.877780 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:12.877821 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.877825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.877873 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.878834 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.878934 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.879792 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.880952 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.881318 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.881772 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.882985 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.882991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.882995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.883033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.890950 (+ 7917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.891998 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.892001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.892002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.892079 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.892410 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.892414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.892415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.892657 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.893365 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.893444 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.893599 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3845,"thread_start_us":87,"threads_started":1}
I20260928 23:44:12.893945 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.874768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.21.112.124:46621)
0928 23:44:12.876251 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.876255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.876551 (+ 296us) server_negotiation.cc:407] Connection header received
0928 23:44:12.876588 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.876590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.876630 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.876697 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:12.877610 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.882009 (+ 4399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.884688 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.885021 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.885105 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.887300 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:12.887303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:12.887306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:12.887327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:12.891879 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.892309 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.892313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.892316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.892402 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:12.892699 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:12.892705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:12.892709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:12.892914 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:12.893057 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.893653 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.893767 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":70,"threads_started":1}
I20260928 23:44:12.897424 28076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:12.900686 28076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:12.903497 29334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: NON_VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: true } } }
I20260928 23:44:12.903676 29334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.907061 29184 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: attempting to promote NON_VOTER 059d32c2c2f249d0ae3a59c33a610737 to VOTER
I20260928 23:44:12.907382 29184 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } }
I20260928 23:44:12.908511 28870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 LEARNER]: Refusing update from remote peer f1b0b16080d14035b8611d345c843120: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 23:44:12.908790 28687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Refusing update from remote peer f1b0b16080d14035b8611d345c843120: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 23:44:12.908988 29334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 23:44:12.909638 29334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 23:44:12.910830 29333 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.911336 28870 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.911754 28687 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.912474 29333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.912573 29333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.912693 29333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.912772 29333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.912873 29333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.912971 29333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:12.913313 29201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.913434 29201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:12.914393 29342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1b0b16080d14035b8611d345c843120" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:12.914521 29342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:12.921948 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.916036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:46149 (local address 127.0.0.1:52642)
0928 23:44:12.916589 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.916608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.916692 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.917814 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.917818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.917837 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:12.917953 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.917959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.919422 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.919426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.920579 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.920589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.921515 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.921519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.921630 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.921653 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.921738 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.921817 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467}
I20260928 23:44:12.922216 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.916869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.21.112.65:46149)
0928 23:44:12.917180 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.917196 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.917218 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:44:12.917303 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.917308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.917642 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.918094 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:12.918207 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.919291 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.920780 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.921319 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.921487 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.921658 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.921770 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.921864 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260928 23:44:12.922437 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.916486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.21.112.66:46087)
0928 23:44:12.916816 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.916826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.917201 (+ 375us) server_negotiation.cc:407] Connection header received
0928 23:44:12.917524 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.917553 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.917803 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.917988 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:12.918216 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.919202 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.920552 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.921379 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.921458 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.921646 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.922193 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.922290 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260928 23:44:12.923384 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.917309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37599 (local address 127.0.0.1:44964)
0928 23:44:12.918216 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.918229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.918318 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.918749 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.918752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.918771 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:12.918905 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.918911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.920325 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.920328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.921515 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.921525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.922985 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.922989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.923071 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.923092 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.923165 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.923237 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":96,"threads_started":1}
I20260928 23:44:12.923436 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.917473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.21.112.67:37599)
0928 23:44:12.917836 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:12.917850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:12.918327 (+ 477us) server_negotiation.cc:407] Connection header received
0928 23:44:12.918409 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:12.918414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:12.918603 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:12.918750 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:12.919018 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.920178 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.921697 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.922704 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.922801 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.922926 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:12.923197 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 23:44:12.923275 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":154,"threads_started":1}
I20260928 23:44:12.923565 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:12.916312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:46087 (local address 127.0.0.1:33394)
0928 23:44:12.917091 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:12.917104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:12.917208 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:12.917950 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:12.917952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:12.917970 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:12.918077 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.918086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.919343 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.919345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.920418 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:12.920427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:12.921939 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:12.921942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:12.922030 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:12.922050 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:44:12.923353 (+ 1303us) client_negotiation.cc:240] Negotiation successful
0928 23:44:12.923430 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":129,"threads_started":1}
I20260928 23:44:12.925810 28973 tablet_service.cc:1467] Processing CreateTablet for tablet b1737ca3af7445cb90b46255dc3b879d (DEFAULT_TABLE table=test-table [id=2aab1b5fd55f4e0c9efbcc1edbae804a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:12.925909 29110 tablet_service.cc:1467] Processing CreateTablet for tablet b1737ca3af7445cb90b46255dc3b879d (DEFAULT_TABLE table=test-table [id=2aab1b5fd55f4e0c9efbcc1edbae804a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:12.926028 29249 tablet_service.cc:1467] Processing CreateTablet for tablet b1737ca3af7445cb90b46255dc3b879d (DEFAULT_TABLE table=test-table [id=2aab1b5fd55f4e0c9efbcc1edbae804a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:12.926997 29110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1737ca3af7445cb90b46255dc3b879d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:12.927002 29249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1737ca3af7445cb90b46255dc3b879d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:12.927325 28973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1737ca3af7445cb90b46255dc3b879d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:12.935367 29348 tablet_bootstrap.cc:492] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Bootstrap starting.
I20260928 23:44:12.937592 29350 tablet_bootstrap.cc:492] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Bootstrap starting.
I20260928 23:44:12.938095 29348 tablet_bootstrap.cc:654] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:12.938323 29349 tablet_bootstrap.cc:492] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Bootstrap starting.
I20260928 23:44:12.938984 29348 log.cc:824] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:12.939735 29350 tablet_bootstrap.cc:654] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:12.940392 29349 tablet_bootstrap.cc:654] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:12.941147 29348 tablet_bootstrap.cc:492] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: No bootstrap required, opened a new log
I20260928 23:44:12.941220 29349 log.cc:824] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:12.941351 29348 ts_tablet_manager.cc:1397] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:44:12.941431 29350 log.cc:824] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:12.943243 29349 tablet_bootstrap.cc:492] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: No bootstrap required, opened a new log
I20260928 23:44:12.943293 29350 tablet_bootstrap.cc:492] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: No bootstrap required, opened a new log
I20260928 23:44:12.943434 29349 ts_tablet_manager.cc:1397] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 23:44:12.943473 29350 ts_tablet_manager.cc:1397] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:44:12.946237 29348 raft_consensus.cc:348] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:12.946621 29348 raft_consensus.cc:374] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:12.946693 29348 raft_consensus.cc:729] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d2645e4057458692cde93aaed70e46, State: Initialized, Role: FOLLOWER
I20260928 23:44:12.947243 29348 consensus_queue.cc:260] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:12.947817 29350 raft_consensus.cc:348] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:12.948102 29350 raft_consensus.cc:374] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:12.948169 29350 raft_consensus.cc:729] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87a2836595e4202906b910e99da3dd1, State: Initialized, Role: FOLLOWER
I20260928 23:44:12.948096 29349 raft_consensus.cc:348] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:12.948339 29349 raft_consensus.cc:374] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:12.948426 29348 ts_tablet_manager.cc:1428] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:44:12.948475 29349 raft_consensus.cc:729] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa63edaf2320442198a32ec812e069b3, State: Initialized, Role: FOLLOWER
I20260928 23:44:12.948645 29350 consensus_queue.cc:260] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:12.948778 29177 heartbeater.cc:503] Master 127.21.112.126:46615 was elected leader, sending a full tablet report...
I20260928 23:44:12.948922 29349 consensus_queue.cc:260] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:12.949620 29350 ts_tablet_manager.cc:1428] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:44:12.949972 29040 heartbeater.cc:503] Master 127.21.112.126:46615 was elected leader, sending a full tablet report...
I20260928 23:44:12.949976 29349 ts_tablet_manager.cc:1428] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260928 23:44:12.951110 29316 heartbeater.cc:503] Master 127.21.112.126:46615 was elected leader, sending a full tablet report...
W20260928 23:44:13.118424 29318 tablet.cc:2369] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:44:13.154161 29179 tablet.cc:2369] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:44:13.171797 29356 raft_consensus.cc:482] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:13.172076 29356 raft_consensus.cc:504] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:13.173393 29356 leader_election.cc:288] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149), 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
I20260928 23:44:13.177289 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.173971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:36787 (local address 127.21.112.65:46149)
0928 23:44:13.174107 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.174110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.174124 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:13.174172 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.174175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.174212 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.174313 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:13.174608 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.175215 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.176347 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.176737 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.176859 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.176913 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.177085 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.177157 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:44:13.177311 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.173707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:46149 (local address 127.21.112.67:36787)
0928 23:44:13.173873 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:13.173889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:13.173997 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:13.174340 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:13.174344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:13.174362 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:13.174496 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:13.174502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.175317 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.175323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:13.176226 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:13.176235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.176852 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.176856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:13.176954 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.176977 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:44:13.177071 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:44:13.177161 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1}
I20260928 23:44:13.177454 29319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.173707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:46087 (local address 127.21.112.67:43693)
0928 23:44:13.174191 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:13.174206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:13.174320 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:13.174576 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:13.174579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:13.174593 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:13.174719 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:13.174724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.175524 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.175527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:13.176336 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:13.176345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.176968 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.176971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:13.177080 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.177097 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:44:13.177183 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:44:13.177257 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":367}
I20260928 23:44:13.177575 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.173831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43693 (local address 127.21.112.66:46087)
0928 23:44:13.173961 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.173965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.174286 (+ 321us) server_negotiation.cc:407] Connection header received
0928 23:44:13.174427 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.174430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.174478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.174556 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:13.174826 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.175409 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.176456 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.176842 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.176968 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.177015 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.177365 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.177438 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:44:13.178082 28993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1737ca3af7445cb90b46255dc3b879d" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87a2836595e4202906b910e99da3dd1" is_pre_election: true
I20260928 23:44:13.178082 29130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1737ca3af7445cb90b46255dc3b879d" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d2645e4057458692cde93aaed70e46" is_pre_election: true
I20260928 23:44:13.178404 28993 raft_consensus.cc:2454] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 0.
I20260928 23:44:13.178404 29130 raft_consensus.cc:2454] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 0.
I20260928 23:44:13.178854 29202 leader_election.cc:302] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68d2645e4057458692cde93aaed70e46, aa63edaf2320442198a32ec812e069b3; no voters:
I20260928 23:44:13.179193 29356 raft_consensus.cc:2781] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:13.179286 29356 raft_consensus.cc:482] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:13.179318 29356 raft_consensus.cc:3037] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:13.180392 29356 raft_consensus.cc:504] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:13.180804 29356 leader_election.cc:288] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 election: Requested vote from peers f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149), 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
I20260928 23:44:13.181213 28993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1737ca3af7445cb90b46255dc3b879d" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87a2836595e4202906b910e99da3dd1"
I20260928 23:44:13.181236 29130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1737ca3af7445cb90b46255dc3b879d" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d2645e4057458692cde93aaed70e46"
I20260928 23:44:13.181353 29130 raft_consensus.cc:3037] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:13.181353 28993 raft_consensus.cc:3037] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:13.182622 29130 raft_consensus.cc:2454] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 1.
I20260928 23:44:13.182622 28993 raft_consensus.cc:2454] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 1.
I20260928 23:44:13.182914 29205 leader_election.cc:302] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa63edaf2320442198a32ec812e069b3, f87a2836595e4202906b910e99da3dd1; no voters:
I20260928 23:44:13.183146 29356 raft_consensus.cc:2781] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:13.183367 29356 raft_consensus.cc:686] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 1 LEADER]: Becoming Leader. State: Replica: aa63edaf2320442198a32ec812e069b3, State: Running, Role: LEADER
I20260928 23:44:13.183772 29356 consensus_queue.cc:237] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } }
I20260928 23:44:13.187600 28075 catalog_manager.cc:5495] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa63edaf2320442198a32ec812e069b3 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "aa63edaf2320442198a32ec812e069b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } health_report { overall_health: HEALTHY } } }
W20260928 23:44:13.195940 29042 tablet.cc:2369] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:44:13.219344 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.215428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.21.112.67:37599)
0928 23:44:13.215584 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.215589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.215603 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:13.215773 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.215776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.215832 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.215940 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:13.216200 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.216701 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.217674 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.217921 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.217999 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.218364 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:13.218821 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:13.218923 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.219038 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.219127 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:44:13.225638 29130 raft_consensus.cc:1261] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Refusing update from remote peer aa63edaf2320442198a32ec812e069b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:13.225672 28993 raft_consensus.cc:1261] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Refusing update from remote peer aa63edaf2320442198a32ec812e069b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:13.226418 29357 consensus_queue.cc:1035] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:13.226648 29356 consensus_queue.cc:1035] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:13.235347 29361 mvcc.cc:205] Tried to move back new op lower bound from 7334457562004668416 to 7334457561843417088. Current Snapshot: MvccSnapshot[applied={T|T < 7334457562004668416}]
I20260928 23:44:13.235385 29360 mvcc.cc:205] Tried to move back new op lower bound from 7334457562004668416 to 7334457561843417088. Current Snapshot: MvccSnapshot[applied={T|T < 7334457562004668416 or (T in {7334457562004668416})}]
I20260928 23:44:13.239152 29362 mvcc.cc:205] Tried to move back new op lower bound from 7334457562004668416 to 7334457561843417088. Current Snapshot: MvccSnapshot[applied={T|T < 7334457562004668416}]
I20260928 23:44:13.245357 28075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:13.245555 28075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:13.253863 28075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1b0b16080d14035b8611d345c843120: Sending DeleteTablet for 3 replicas of tablet b1737ca3af7445cb90b46255dc3b879d
I20260928 23:44:13.254940 29110 tablet_service.cc:1514] Processing DeleteTablet for tablet b1737ca3af7445cb90b46255dc3b879d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:13 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:33394
I20260928 23:44:13.254930 28973 tablet_service.cc:1514] Processing DeleteTablet for tablet b1737ca3af7445cb90b46255dc3b879d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:13 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:52642
I20260928 23:44:13.255148 29249 tablet_service.cc:1514] Processing DeleteTablet for tablet b1737ca3af7445cb90b46255dc3b879d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:13 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:44964
I20260928 23:44:13.255597 29369 tablet_replica.cc:330] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: stopping tablet replica
I20260928 23:44:13.255795 29370 tablet_replica.cc:330] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: stopping tablet replica
I20260928 23:44:13.255990 29369 raft_consensus.cc:2229] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:13.256274 29370 raft_consensus.cc:2229] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:13.256446 29369 raft_consensus.cc:2258] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:13.256855 29370 raft_consensus.cc:2258] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:13.257308 29371 tablet_replica.cc:330] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: stopping tablet replica
I20260928 23:44:13.257598 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28044
I20260928 23:44:13.258453 29371 raft_consensus.cc:2229] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:13.259048 29371 raft_consensus.cc:2258] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:13.259950 29369 ts_tablet_manager.cc:1905] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:13.260826 29371 ts_tablet_manager.cc:1905] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:13.262459 29370 ts_tablet_manager.cc:1905] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:13.263712 29371 ts_tablet_manager.cc:1918] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:13.263806 29371 log.cc:1196] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/b1737ca3af7445cb90b46255dc3b879d
I20260928 23:44:13.264137 29371 ts_tablet_manager.cc:1939] T b1737ca3af7445cb90b46255dc3b879d P aa63edaf2320442198a32ec812e069b3: Deleting consensus metadata
I20260928 23:44:13.265859 29369 ts_tablet_manager.cc:1918] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:13.265980 29369 log.cc:1196] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/b1737ca3af7445cb90b46255dc3b879d
I20260928 23:44:13.266336 29369 ts_tablet_manager.cc:1939] T b1737ca3af7445cb90b46255dc3b879d P 68d2645e4057458692cde93aaed70e46: Deleting consensus metadata
W20260928 23:44:13.266726 29202 connection.cc:441] server connection from 127.0.0.1:44964 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44964 (request call id 1) could send its response
I20260928 23:44:13.267442 29370 ts_tablet_manager.cc:1918] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 23:44:13.267536 29063 connection.cc:441] server connection from 127.0.0.1:33394 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33394 (request call id 1) could send its response
I20260928 23:44:13.267671 29370 log.cc:1196] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/b1737ca3af7445cb90b46255dc3b879d
I20260928 23:44:13.268072 29370 ts_tablet_manager.cc:1939] T b1737ca3af7445cb90b46255dc3b879d P f87a2836595e4202906b910e99da3dd1: Deleting consensus metadata
W20260928 23:44:13.269209 28926 connection.cc:441] server connection from 127.0.0.1:52642 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 1) could send its response
I20260928 23:44:13.279345 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.256486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.21.112.125:37867)
0928 23:44:13.256636 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.256641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.267822 (+ 11181us) server_negotiation.cc:407] Connection header received
0928 23:44:13.268041 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.268045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.268101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.268154 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:13.269233 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.270097 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.271313 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.271690 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.271766 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.273740 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:13.273744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:13.273747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:13.273773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:13.276203 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.276731 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.276735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.276737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.276817 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.277120 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.277123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.277125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.277325 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:13.277424 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.277835 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.279158 (+ 1323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:44:13.280980 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.270848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.21.112.124:46621)
0928 23:44:13.270961 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.270966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.271347 (+ 381us) server_negotiation.cc:407] Connection header received
0928 23:44:13.271578 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.271584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.271642 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.271699 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:13.272577 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.273384 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.274664 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.275004 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.275071 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.276940 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:13.276943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:13.276947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:13.276976 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:13.278850 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.279385 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.279392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.279393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.279499 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.279820 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.279825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.279827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.280065 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:13.280189 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.280686 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.280805 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:44:13.281872 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.270194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.21.112.125:37867)
0928 23:44:13.270317 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.270321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.271068 (+ 747us) server_negotiation.cc:407] Connection header received
0928 23:44:13.271252 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.271255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.271311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.271368 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:13.273055 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.273786 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.274963 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.275291 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.275361 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.276985 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:13.276988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:13.276990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:13.277015 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:13.279032 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.280219 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.280225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.280227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.280315 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.280633 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.280636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.280639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.280838 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:13.280936 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.281610 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.281739 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:44:13.287842 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:13.256556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.21.112.124:46621)
0928 23:44:13.256741 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:13.256745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:13.278396 (+ 21651us) server_negotiation.cc:407] Connection header received
0928 23:44:13.278557 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:13.278560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:13.278611 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:13.278659 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:13.279429 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.280138 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.281343 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:13.281755 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:13.281831 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:13.283800 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:13.283803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:13.283805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:13.283828 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:13.285778 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.286330 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.286334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.286335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.286428 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:13.286802 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:13.286805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:13.286807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:13.287052 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:13.287190 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:13.287532 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 23:44:13.287673 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 23:44:14.240134 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.239633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:54221)
0928 23:44:14.239940 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.240021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
W20260928 23:44:14.240490 29177 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:46615 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.241482 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.241008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:41169)
0928 23:44:14.241298 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.241383 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":73,"threads_started":1}
W20260928 23:44:14.241816 29316 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:46615 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.242291 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.241424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:50475)
0928 23:44:14.242143 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.242217 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":60,"threads_started":1}
W20260928 23:44:14.242523 29040 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:46615 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.750597 29382 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1b0b16080d14035b8611d345c843120)
I20260928 23:44:14.750765 29383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f1b0b16080d14035b8611d345c843120)
I20260928 23:44:14.750835 29382 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } }
I20260928 23:44:14.750985 29383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } }
I20260928 23:44:14.751948 29382 leader_election.cc:288] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615), 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124:46621)
I20260928 23:44:14.752315 29383 leader_election.cc:288] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615), e0fb094665de4669a706ba2fd87a3c69 (127.21.112.125:37867)
I20260928 23:44:14.752693 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.752107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34344)
0928 23:44:14.752491 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.752583 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":74,"threads_started":1}
W20260928 23:44:14.752887 28844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.753748 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.752609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34360)
0928 23:44:14.753316 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.753385 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260928 23:44:14.753926 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.753590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34370)
0928 23:44:14.753683 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.753732 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 23:44:14.753921 28661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
W20260928 23:44:14.754285 28844 leader_election.cc:334] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615): Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.755059 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.754828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34386)
0928 23:44:14.754948 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.754999 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 23:44:14.755262 28661 leader_election.cc:334] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615): Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.756659 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.752195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:46621 (local address 127.0.0.1:47390)
0928 23:44:14.752400 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.752416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:14.752524 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:14.752983 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:14.752986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:14.753010 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:14.753118 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.753124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.754302 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.754305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.755405 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.755416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.756191 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.756195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.756347 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.756368 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:44:14.756439 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:44:14.756497 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":42,"thread_start_us":60,"threads_started":1}
I20260928 23:44:14.756711 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.752339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.21.112.124:46621)
0928 23:44:14.752707 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:14.752713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:14.752730 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:14.752778 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:14.752781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:14.752829 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:14.753245 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:14.753346 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.754184 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.755543 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.756029 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.756146 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.756195 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:14.756482 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 23:44:14.756571 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":53,"threads_started":1}
I20260928 23:44:14.757056 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.752180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:37867 (local address 127.0.0.1:36500)
0928 23:44:14.752829 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.752841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:14.752914 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:14.753530 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:14.753556 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:14.753575 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:14.753654 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.753660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.754635 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.754638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.755773 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.755783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.756635 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.756639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.756744 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.756771 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:44:14.756844 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:44:14.756922 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":45,"threads_started":1}
I20260928 23:44:14.757122 28870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0fb094665de4669a706ba2fd87a3c69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "059d32c2c2f249d0ae3a59c33a610737" is_pre_election: true
I20260928 23:44:14.757337 28870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0fb094665de4669a706ba2fd87a3c69 in term 1.
I20260928 23:44:14.757416 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.752798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.21.112.125:37867)
0928 23:44:14.753320 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:14.753327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:14.753341 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:14.753384 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:14.753386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:14.753430 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:14.753524 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:14.753783 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.754510 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.755891 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.756506 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.756616 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.756656 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:14.757219 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 23:44:14.757299 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":473,"thread_start_us":56,"threads_started":1}
I20260928 23:44:14.757700 28662 leader_election.cc:302] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 059d32c2c2f249d0ae3a59c33a610737, e0fb094665de4669a706ba2fd87a3c69; no voters: f1b0b16080d14035b8611d345c843120
I20260928 23:44:14.757859 28687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "059d32c2c2f249d0ae3a59c33a610737" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0fb094665de4669a706ba2fd87a3c69" is_pre_election: true
I20260928 23:44:14.757939 29382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 23:44:14.757988 29382 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f1b0b16080d14035b8611d345c843120)
I20260928 23:44:14.758033 29382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:44:14.759044 29382 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } }
I20260928 23:44:14.759294 28687 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 059d32c2c2f249d0ae3a59c33a610737 in current term 2: Already voted for candidate e0fb094665de4669a706ba2fd87a3c69 in this term.
I20260928 23:44:14.759466 29382 leader_election.cc:288] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [CANDIDATE]: Term 2 election: Requested vote from peers f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615), 059d32c2c2f249d0ae3a59c33a610737 (127.21.112.124:46621)
I20260928 23:44:14.759809 28870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0fb094665de4669a706ba2fd87a3c69" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "059d32c2c2f249d0ae3a59c33a610737"
I20260928 23:44:14.759907 28870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:44:14.760874 28870 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0fb094665de4669a706ba2fd87a3c69 in term 2.
I20260928 23:44:14.761227 28662 leader_election.cc:302] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059d32c2c2f249d0ae3a59c33a610737, e0fb094665de4669a706ba2fd87a3c69; no voters:
I20260928 23:44:14.761310 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.759583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34402)
0928 23:44:14.761194 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.761249 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 23:44:14.761329 28845 leader_election.cc:302] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 059d32c2c2f249d0ae3a59c33a610737; no voters: e0fb094665de4669a706ba2fd87a3c69, f1b0b16080d14035b8611d345c843120
I20260928 23:44:14.761456 29382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:44:14.761566 29383 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:44:14.761686 29382 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [term 2 LEADER]: Becoming Leader. State: Replica: e0fb094665de4669a706ba2fd87a3c69, State: Running, Role: LEADER
I20260928 23:44:14.761979 29382 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } }
I20260928 23:44:14.762079 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.761852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34404)
0928 23:44:14.761959 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.762013 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:44:14.762179 28661 leader_election.cc:334] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615): Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
I20260928 23:44:14.763048 29382 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fb094665de4669a706ba2fd87a3c69. Latest consensus state: current_term: 2 leader_uuid: "e0fb094665de4669a706ba2fd87a3c69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:14.763149 29382 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:14.763486 29394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:14.765879 29394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:14.766232 29394 catalog_manager.cc:1259] Loaded cluster ID: 9a93cd82a70147caaac56fd3a8a59bcf
I20260928 23:44:14.766273 29394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:14.766764 29394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:14.767113 29394 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Loaded TSK: 0
I20260928 23:44:14.767848 29394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:44:14.809710 28676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:14.812800 28676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:14.815846 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.815566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34422)
0928 23:44:14.815702 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.815765 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 23:44:14.815929 28870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [term 2 FOLLOWER]: Refusing update from remote peer e0fb094665de4669a706ba2fd87a3c69: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 23:44:14.816426 29392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 23:44:14.816668 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.816432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.0.0.1:34424)
0928 23:44:14.816533 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.816588 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 23:44:14.816753 28661 consensus_peers.cc:487] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 -> Peer f1b0b16080d14035b8611d345c843120 (127.21.112.126:46615): Couldn't send request to peer f1b0b16080d14035b8611d345c843120. Status: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:44:14.818063 29383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fb094665de4669a706ba2fd87a3c69. Latest consensus state: current_term: 2 leader_uuid: "e0fb094665de4669a706ba2fd87a3c69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:14.818219 29383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:14.819211 29392 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e0fb094665de4669a706ba2fd87a3c69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:14.819339 29392 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:14.819603 29383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e0fb094665de4669a706ba2fd87a3c69" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f1b0b16080d14035b8611d345c843120" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46615 } } peers { permanent_uuid: "e0fb094665de4669a706ba2fd87a3c69" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 37867 } attrs { promote: false } } peers { permanent_uuid: "059d32c2c2f249d0ae3a59c33a610737" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 46621 } attrs { promote: false } } }
I20260928 23:44:14.819736 29383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059d32c2c2f249d0ae3a59c33a610737 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:44:14.830863 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.826182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:46149 (local address 127.0.0.1:59008)
0928 23:44:14.826324 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.826341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:14.826417 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:14.827274 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:14.827280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:14.827302 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:14.827406 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.827424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.828464 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.828470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.829596 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.829605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.830488 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.830492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.830578 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.830605 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:44:14.830650 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:44:14.830720 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":24}
I20260928 23:44:14.831175 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.826448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37599 (local address 127.0.0.1:53670)
0928 23:44:14.826967 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.826980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:14.827055 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:14.827694 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:14.827697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:14.827715 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:14.827795 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.827800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.828786 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.828789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.829962 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.829970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.830759 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.830762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.830849 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.830869 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:44:14.830956 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:44:14.831042 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":119}
I20260928 23:44:14.831411 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.826728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.21.112.65:46149)
0928 23:44:14.826993 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:14.826998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:14.827017 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:44:14.827076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:14.827079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:14.827166 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:14.827250 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:14.827565 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.828361 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.829729 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.830369 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.831077 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.831152 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:14.831200 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:44:14.831280 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
I20260928 23:44:14.831465 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.826721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:46087 (local address 127.0.0.1:45706)
0928 23:44:14.826991 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.827007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:14.827096 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:14.827909 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:14.827911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:14.827925 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:14.827997 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.828002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.828971 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.828975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.830152 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:14.830161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.831132 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.831135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:14.831227 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.831247 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:44:14.831291 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:44:14.831350 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260928 23:44:14.831748 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.826573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.21.112.67:37599)
0928 23:44:14.827467 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:14.827474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:14.827488 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:14.827535 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:14.827538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:14.827583 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:14.827674 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:14.827911 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.828658 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.830098 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.830635 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.830760 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.830823 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:14.831176 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 23:44:14.831435 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":93,"threads_started":1}
I20260928 23:44:14.831998 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.827117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.21.112.66:46087)
0928 23:44:14.827458 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:14.827463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:14.827476 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:44:14.827527 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:14.827530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:14.827576 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:14.827674 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:14.828107 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.828839 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.830284 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:14.830909 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:14.831703 (+ 794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:14.831752 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:14.831781 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 23:44:14.831881 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":162,"threads_started":1}
I20260928 23:44:14.832063 28973 tablet_service.cc:1467] Processing CreateTablet for tablet 6218f9ef2c984b03a765f374152c2f21 (DEFAULT_TABLE table=test-table [id=0637c23756ab446797a38a1a1233af90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:14.832209 29249 tablet_service.cc:1467] Processing CreateTablet for tablet 6218f9ef2c984b03a765f374152c2f21 (DEFAULT_TABLE table=test-table [id=0637c23756ab446797a38a1a1233af90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:14.832365 28973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6218f9ef2c984b03a765f374152c2f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:14.832463 29249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6218f9ef2c984b03a765f374152c2f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:14.832458 29110 tablet_service.cc:1467] Processing CreateTablet for tablet 6218f9ef2c984b03a765f374152c2f21 (DEFAULT_TABLE table=test-table [id=0637c23756ab446797a38a1a1233af90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:14.832700 29110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6218f9ef2c984b03a765f374152c2f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:14.834432 29405 tablet_bootstrap.cc:492] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1: Bootstrap starting.
I20260928 23:44:14.835009 29406 tablet_bootstrap.cc:492] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Bootstrap starting.
I20260928 23:44:14.835371 29405 tablet_bootstrap.cc:654] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:14.835865 29407 tablet_bootstrap.cc:492] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Bootstrap starting.
I20260928 23:44:14.836055 29406 tablet_bootstrap.cc:654] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:14.836728 29407 tablet_bootstrap.cc:654] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:14.837102 29406 tablet_bootstrap.cc:492] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: No bootstrap required, opened a new log
I20260928 23:44:14.837217 29406 ts_tablet_manager.cc:1397] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:14.837662 29406 raft_consensus.cc:348] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:14.837774 29406 raft_consensus.cc:374] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:14.837818 29406 raft_consensus.cc:729] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa63edaf2320442198a32ec812e069b3, State: Initialized, Role: FOLLOWER
I20260928 23:44:14.837860 29407 tablet_bootstrap.cc:492] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: No bootstrap required, opened a new log
I20260928 23:44:14.837945 29407 ts_tablet_manager.cc:1397] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:14.837955 29406 consensus_queue.cc:260] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:14.838303 29406 ts_tablet_manager.cc:1428] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:14.838320 29407 raft_consensus.cc:348] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:14.838435 29407 raft_consensus.cc:374] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:14.838465 29407 raft_consensus.cc:729] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d2645e4057458692cde93aaed70e46, State: Initialized, Role: FOLLOWER
I20260928 23:44:14.838580 29407 consensus_queue.cc:260] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:14.838938 29407 ts_tablet_manager.cc:1428] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:14.839190 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.838745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:46077)
0928 23:44:14.839036 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.839099 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260928 23:44:14.840443 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.840046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:52881)
0928 23:44:14.840278 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.840337 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260928 23:44:14.840826 29317 heartbeater.cc:503] Master 127.21.112.125:37867 was elected leader, sending a full tablet report...
I20260928 23:44:14.841219 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.840936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:41781)
0928 23:44:14.841099 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.841151 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:44:14.841909 29405 tablet_bootstrap.cc:492] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1: No bootstrap required, opened a new log
I20260928 23:44:14.842016 29405 ts_tablet_manager.cc:1397] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 23:44:14.842033 29178 heartbeater.cc:503] Master 127.21.112.125:37867 was elected leader, sending a full tablet report...
I20260928 23:44:14.842191 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.841826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:33205)
0928 23:44:14.842060 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.842113 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 23:44:14.842505 29405 raft_consensus.cc:348] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:14.842604 29405 raft_consensus.cc:374] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:14.842626 29405 raft_consensus.cc:729] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87a2836595e4202906b910e99da3dd1, State: Initialized, Role: FOLLOWER
I20260928 23:44:14.842757 29405 consensus_queue.cc:260] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:14.843088 29405 ts_tablet_manager.cc:1428] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:44:14.843866 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.843327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:42239)
0928 23:44:14.843718 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.843775 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":94,"threads_started":1}
I20260928 23:44:14.844645 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:14.844293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:50369)
0928 23:44:14.844527 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:14.844581 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 23:44:14.844691 29041 heartbeater.cc:503] Master 127.21.112.125:37867 was elected leader, sending a full tablet report...
I20260928 23:44:15.044919 29411 raft_consensus.cc:482] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:15.045110 29411 raft_consensus.cc:504] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:15.045653 29411 leader_election.cc:288] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149), 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
I20260928 23:44:15.046169 28993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6218f9ef2c984b03a765f374152c2f21" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87a2836595e4202906b910e99da3dd1" is_pre_election: true
I20260928 23:44:15.046226 29130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6218f9ef2c984b03a765f374152c2f21" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d2645e4057458692cde93aaed70e46" is_pre_election: true
I20260928 23:44:15.046329 28993 raft_consensus.cc:2454] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 0.
I20260928 23:44:15.046468 29130 raft_consensus.cc:2454] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 0.
I20260928 23:44:15.046802 29205 leader_election.cc:302] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa63edaf2320442198a32ec812e069b3, f87a2836595e4202906b910e99da3dd1; no voters:
I20260928 23:44:15.047016 29411 raft_consensus.cc:2781] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:15.047075 29411 raft_consensus.cc:482] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:15.047139 29411 raft_consensus.cc:3037] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:15.048285 29411 raft_consensus.cc:504] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:15.048669 29411 leader_election.cc:288] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 election: Requested vote from peers f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149), 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087)
I20260928 23:44:15.049013 28993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6218f9ef2c984b03a765f374152c2f21" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f87a2836595e4202906b910e99da3dd1"
I20260928 23:44:15.049134 28993 raft_consensus.cc:3037] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:15.049168 29130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6218f9ef2c984b03a765f374152c2f21" candidate_uuid: "aa63edaf2320442198a32ec812e069b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d2645e4057458692cde93aaed70e46"
I20260928 23:44:15.049328 29130 raft_consensus.cc:3037] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:15.050189 28993 raft_consensus.cc:2454] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 1.
I20260928 23:44:15.050472 29205 leader_election.cc:302] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa63edaf2320442198a32ec812e069b3, f87a2836595e4202906b910e99da3dd1; no voters:
I20260928 23:44:15.050465 29130 raft_consensus.cc:2454] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa63edaf2320442198a32ec812e069b3 in term 1.
I20260928 23:44:15.050622 29411 raft_consensus.cc:2781] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:15.050863 29411 raft_consensus.cc:686] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 1 LEADER]: Becoming Leader. State: Replica: aa63edaf2320442198a32ec812e069b3, State: Running, Role: LEADER
I20260928 23:44:15.051034 29411 consensus_queue.cc:237] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } }
I20260928 23:44:15.051604 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.051348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:51717)
0928 23:44:15.051456 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.051525 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:44:15.053022 28676 catalog_manager.cc:5495] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa63edaf2320442198a32ec812e069b3 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "aa63edaf2320442198a32ec812e069b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa63edaf2320442198a32ec812e069b3" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 } health_report { overall_health: UNKNOWN } } }
I20260928 23:44:15.071089 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.068213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.21.112.67:37599)
0928 23:44:15.068345 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:15.068349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:15.068363 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:15.068410 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:15.068412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:15.068455 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:15.068526 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:15.068810 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:15.069363 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:15.070210 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:15.070418 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:15.070497 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:15.070719 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:15.070789 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:15.070840 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:15.070928 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:44:15.070978 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:44:15.073329 29130 raft_consensus.cc:1261] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Refusing update from remote peer aa63edaf2320442198a32ec812e069b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:15.073329 28993 raft_consensus.cc:1261] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Refusing update from remote peer aa63edaf2320442198a32ec812e069b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:15.073794 29411 consensus_queue.cc:1035] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87a2836595e4202906b910e99da3dd1" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:15.073956 29417 consensus_queue.cc:1035] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d2645e4057458692cde93aaed70e46" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:15.075682 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.075427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:52777)
0928 23:44:15.075523 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.075587 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:44:15.075825 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.075572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:42805)
0928 23:44:15.075702 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.075754 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 23:44:15.079912 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.079607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.66:49827)
0928 23:44:15.079762 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.079819 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 23:44:15.080588 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.079748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.67:50105)
0928 23:44:15.080457 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.080513 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
I20260928 23:44:15.080952 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.080641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46615 (local address 127.21.112.65:36975)
0928 23:44:15.080830 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.080882 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 23:44:15.084151 28676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:15.084321 28676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:15.088236 28676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0fb094665de4669a706ba2fd87a3c69: Sending DeleteTablet for 3 replicas of tablet 6218f9ef2c984b03a765f374152c2f21
I20260928 23:44:15.089210 28973 tablet_service.cc:1514] Processing DeleteTablet for tablet 6218f9ef2c984b03a765f374152c2f21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:15 UTC) from {username='oryx', principal='oryx/127.21.112.125@KRBTEST.COM'} at 127.0.0.1:59008
I20260928 23:44:15.089222 29110 tablet_service.cc:1514] Processing DeleteTablet for tablet 6218f9ef2c984b03a765f374152c2f21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:15 UTC) from {username='oryx', principal='oryx/127.21.112.125@KRBTEST.COM'} at 127.0.0.1:45706
I20260928 23:44:15.089231 29249 tablet_service.cc:1514] Processing DeleteTablet for tablet 6218f9ef2c984b03a765f374152c2f21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:15 UTC) from {username='oryx', principal='oryx/127.21.112.125@KRBTEST.COM'} at 127.0.0.1:53670
I20260928 23:44:15.089632 29431 tablet_replica.cc:330] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: stopping tablet replica
I20260928 23:44:15.089706 29430 tablet_replica.cc:330] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1: stopping tablet replica
I20260928 23:44:15.089771 29431 raft_consensus.cc:2229] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:15.089825 29430 raft_consensus.cc:2229] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:15.089910 29431 raft_consensus.cc:2258] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:15.089962 29430 raft_consensus.cc:2258] T 6218f9ef2c984b03a765f374152c2f21 P f87a2836595e4202906b910e99da3dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:15.090101 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28821
I20260928 23:44:15.091320 29431 ts_tablet_manager.cc:1905] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:15.094467 29431 ts_tablet_manager.cc:1918] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:15.094534 29431 log.cc:1196] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/6218f9ef2c984b03a765f374152c2f21
I20260928 23:44:15.094823 29431 ts_tablet_manager.cc:1939] T 6218f9ef2c984b03a765f374152c2f21 P 68d2645e4057458692cde93aaed70e46: Deleting consensus metadata
I20260928 23:44:15.094985 29428 tablet_replica.cc:330] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: stopping tablet replica
I20260928 23:44:15.095153 29428 raft_consensus.cc:2229] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:15.095378 29428 raft_consensus.cc:2258] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:15.095674 28661 catalog_manager.cc:4841] TS 68d2645e4057458692cde93aaed70e46 (127.21.112.66:46087): tablet 6218f9ef2c984b03a765f374152c2f21 (table test-table [id=0637c23756ab446797a38a1a1233af90]) successfully deleted
I20260928 23:44:15.096101 29428 ts_tablet_manager.cc:1905] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:15.098135 29428 ts_tablet_manager.cc:1918] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:15.098233 29428 log.cc:1196] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/6218f9ef2c984b03a765f374152c2f21
I20260928 23:44:15.098547 29428 ts_tablet_manager.cc:1939] T 6218f9ef2c984b03a765f374152c2f21 P aa63edaf2320442198a32ec812e069b3: Deleting consensus metadata
I20260928 23:44:15.099009 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 29047
I20260928 23:44:15.099112 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:15.098803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:46149 (local address 127.0.0.1:59018)
0928 23:44:15.098952 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:15.099010 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:46149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 23:44:15.099720 28664 catalog_manager.cc:4584] TS f87a2836595e4202906b910e99da3dd1 (127.21.112.65:46149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6218f9ef2c984b03a765f374152c2f21: Network error: Client connection negotiation failed: client connection to 127.21.112.65:46149: connect: Connection refused (error 111)
I20260928 23:44:15.099880 28664 catalog_manager.cc:4841] TS aa63edaf2320442198a32ec812e069b3 (127.21.112.67:37599): tablet 6218f9ef2c984b03a765f374152c2f21 (table test-table [id=0637c23756ab446797a38a1a1233af90]) successfully deleted
I20260928 23:44:15.105616 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 29183
I20260928 23:44:15.113312 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28540
I20260928 23:44:15.119352 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 28649
2026-09-28T23:44:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5722 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:15 dist-test-slave-c3s8 krb5kdc[29434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:15 dist-test-slave-c3s8 krb5kdc[29434](info): set up 2 sockets
Sep 28 23:44:15 dist-test-slave-c3s8 krb5kdc[29434](info): commencing operation
krb5kdc: starting...
W20260928 23:44:16.172813 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:16 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab.
Entry for principal oryx/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab.
I20260928 23:44:16.309212 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:44619
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab
--principal=oryx/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:44619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:16.421012 29447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:16.421317 29447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:16.421396 29447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:16.425323 29447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:16.425411 29447 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:44:16.425444 29447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:16.425487 29447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:16.425522 29447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:16.425577 29447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:16.430884 29447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:44619
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.126.keytab
--principal=oryx/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:44619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:16.431998 29447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:16.433076 29447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:16.441916 29453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:16.442019 29455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:16.442168 29452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:16.442224 29447 server_base.cc:1034] running on GCE node
I20260928 23:44:16.442961 29447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:16.444329 29447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:16.445585 29447 hybrid_clock.cc:648] HybridClock initialized: now 1790639056445557 us; error 42 us; skew 500 ppm
Sep 28 23:44:16 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:16.448423 29447 init.cc:377] Logged in from keytab as oryx/127.21.112.126@KRBTEST.COM (short username oryx)
I20260928 23:44:16.449312 29447 webserver.cc:460] Webserver started at http://127.21.112.126:46003/ using document root <none> and password file <none>
I20260928 23:44:16.449925 29447 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:16.449990 29447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:16.450157 29447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:16.452292 29447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "d6396f2b38db4df187754ef81d9dc3cf"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "8e6a13a98133316ed3097f98f1c09411"
server_key_iv: "be6adfd24c2def63218476f66781c5c6"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.452816 29447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "d6396f2b38db4df187754ef81d9dc3cf"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "8e6a13a98133316ed3097f98f1c09411"
server_key_iv: "be6adfd24c2def63218476f66781c5c6"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.456753 29447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:44:16.459283 29462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:16.460588 29447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:16.460700 29447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "d6396f2b38db4df187754ef81d9dc3cf"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "8e6a13a98133316ed3097f98f1c09411"
server_key_iv: "be6adfd24c2def63218476f66781c5c6"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.460793 29447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:16.485081 29447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:16.488894 29447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:16.489073 29447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:16.497299 29447 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:44619
I20260928 23:44:16.497318 29525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:44619 every 8 connection(s)
I20260928 23:44:16.498301 29447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:16.501019 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:16.505475 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 29447
I20260928 23:44:16.505677 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:16.505981 21953 external_mini_cluster.cc:1428] Setting key a4403983ab191b44f92355b2dbeabe3b
I20260928 23:44:16.506768 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Bootstrap starting.
I20260928 23:44:16.509431 29526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:16.510345 29526 log.cc:824] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:16.512414 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: No bootstrap required, opened a new log
Sep 28 23:44:16 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:16.516870 29526 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6396f2b38db4df187754ef81d9dc3cf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44619 } }
I20260928 23:44:16.517191 29526 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:16.517264 29526 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6396f2b38db4df187754ef81d9dc3cf, State: Initialized, Role: FOLLOWER
I20260928 23:44:16.517751 29526 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6396f2b38db4df187754ef81d9dc3cf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44619 } }
I20260928 23:44:16.517913 29526 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:16.518000 29526 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:16.518100 29526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:16.519305 29526 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6396f2b38db4df187754ef81d9dc3cf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44619 } }
I20260928 23:44:16.519645 29526 leader_election.cc:302] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6396f2b38db4df187754ef81d9dc3cf; no voters:
I20260928 23:44:16.519996 29526 leader_election.cc:288] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:16.520131 29531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:16.520540 29531 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [term 1 LEADER]: Becoming Leader. State: Replica: d6396f2b38db4df187754ef81d9dc3cf, State: Running, Role: LEADER
I20260928 23:44:16.520898 29531 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6396f2b38db4df187754ef81d9dc3cf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44619 } }
I20260928 23:44:16.521309 29526 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:16.521353 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:16.507478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.21.112.126:44619)
0928 23:44:16.507979 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:16.507996 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:16.508044 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:44:16.508876 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:16.508899 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:16.509326 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:16.509745 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:16.510377 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.511782 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.512677 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.513363 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.513525 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:16.516254 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:16.516261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:16.516279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:16.516311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:16.518350 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:16.518810 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:16.518814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:16.518819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:16.518928 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:16.519235 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:16.519237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:16.519238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:16.519672 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:16.519868 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:16.520175 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 23:44:16.520463 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260928 23:44:16.522559 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6396f2b38db4df187754ef81d9dc3cf. Latest consensus state: current_term: 1 leader_uuid: "d6396f2b38db4df187754ef81d9dc3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6396f2b38db4df187754ef81d9dc3cf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44619 } } }
I20260928 23:44:16.522706 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:16.523032 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6396f2b38db4df187754ef81d9dc3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6396f2b38db4df187754ef81d9dc3cf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44619 } } }
I20260928 23:44:16.523129 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf [sys.catalog]: This master's current role is: LEADER
W20260928 23:44:16.526472 29546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:44:16.526551 29546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:44:16.526679 29541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:16.527488 29541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:16.532807 29541 catalog_manager.cc:1347] Generated new cluster ID: 4c21f062abca49ec9b02c33ca25a8015
I20260928 23:44:16.532882 29541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:16.552345 29541 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:16.553737 29541 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:16.565464 29541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Generated new TSK 0
I20260928 23:44:16.566295 29541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab.
Entry for principal oryx/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab.
I20260928 23:44:16.596325 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44619
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:16.711130 29552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:16.711417 29552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:16.711463 29552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:16.715739 29552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:16.715806 29552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:16.715847 29552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:44:16.715909 29552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:16.721292 29552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:16.722553 29552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:16.723656 29552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:16.733449 29558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:16.733443 29560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:16.733603 29557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:16.733726 29552 server_base.cc:1034] running on GCE node
I20260928 23:44:16.734213 29552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:16.734807 29552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:16.735988 29552 hybrid_clock.cc:648] HybridClock initialized: now 1790639056735966 us; error 30 us; skew 500 ppm
Sep 28 23:44:16 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:16.739080 29552 init.cc:377] Logged in from keytab as oryx/127.21.112.65@KRBTEST.COM (short username oryx)
I20260928 23:44:16.740132 29552 webserver.cc:460] Webserver started at http://127.21.112.65:34037/ using document root <none> and password file <none>
I20260928 23:44:16.740784 29552 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:16.740883 29552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:16.741122 29552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:16.743304 29552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "bd8600defd184efeb8444be2038f0ac2"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "11eeb13362dcb69233456e81e1d1d216"
server_key_iv: "d10361b742a3455ce73b4f8ef60ab594"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.743868 29552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "bd8600defd184efeb8444be2038f0ac2"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "11eeb13362dcb69233456e81e1d1d216"
server_key_iv: "d10361b742a3455ce73b4f8ef60ab594"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.747784 29552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 23:44:16.750588 29567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:16.751852 29552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:16.751993 29552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "bd8600defd184efeb8444be2038f0ac2"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "11eeb13362dcb69233456e81e1d1d216"
server_key_iv: "d10361b742a3455ce73b4f8ef60ab594"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.752116 29552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:16.781442 29552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:16.784934 29552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:16.785152 29552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:16.786711 29552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:16.786798 29552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:16.786870 29552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:16.786906 29552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:44:16 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:16.798200 29552 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:35211
I20260928 23:44:16.799019 29682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:35211 every 8 connection(s)
I20260928 23:44:16.799274 29552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:16.800789 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:16.787760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45949 (local address 127.21.112.126:44619)
0928 23:44:16.787888 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:16.787891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:16.788987 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 23:44:16.790230 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:16.790234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:16.790291 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:16.790405 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:16.791691 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.792223 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.793469 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.793860 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.793990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:16.796873 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:16.796876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:16.796878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:16.796902 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:16.798580 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:16.799203 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:16.799206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:16.799208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:16.799294 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:16.799645 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:16.799648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:16.799650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:16.799873 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:16.799967 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:16.800542 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 23:44:16.800652 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:44:16.801610 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:16.788341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.21.112.65:45949)
0928 23:44:16.788833 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:16.788876 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:16.789949 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:16.790663 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:16.790674 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:16.791051 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:16.791538 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:16.791551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.792350 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.792354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:16.793345 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:16.793354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.794042 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.794046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:16.794217 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:16.794859 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:16.794883 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:16.796696 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:16.798703 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:16.798710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:16.798726 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:16.799077 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:16.799393 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:16.799396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:16.799398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:16.799543 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:16.800019 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:16.800025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:16.800304 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 23:44:16.800464 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 23:44:16.800716 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
I20260928 23:44:16.802766 29683 heartbeater.cc:347] Connected to a master server at 127.21.112.126:44619
I20260928 23:44:16.803002 29683 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:16.803264 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 29552
I20260928 23:44:16.803396 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:16.803560 29683 heartbeater.cc:511] Master 127.21.112.126:44619 requested a full tablet report, sending...
I20260928 23:44:16.803758 21953 external_mini_cluster.cc:1428] Setting key 3bc49b1948f69cb8196f44abcbfbf83c
I20260928 23:44:16.805105 29479 ts_manager.cc:194] Registered new tserver with Master: bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211)
I20260928 23:44:16.806443 29479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:45949
I20260928 23:44:16.812716 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:16.805319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:48425 (local address 127.21.112.126:44619)
0928 23:44:16.805426 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:16.805430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:16.805715 (+ 285us) server_negotiation.cc:407] Connection header received
0928 23:44:16.805825 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:16.805828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:16.805872 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:16.805954 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:16.806522 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.807215 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.808035 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.808272 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.808379 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:16.809641 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:16.809644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:16.809646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:16.809666 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:16.811143 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:16.811642 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:16.811644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:16.811645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:16.811712 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:16.812005 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:16.812008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:16.812010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:16.812161 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:16.812230 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:16.812527 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 23:44:16.812611 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:44:16.813005 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:16.805341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.21.112.65:48425)
0928 23:44:16.805619 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:16.805634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:16.805731 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:16.805990 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:16.805996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:16.806238 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:16.806393 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:16.806404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.807341 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.807344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:16.807923 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:16.807930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:16.808382 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:16.808385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:16.808478 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:16.808973 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:16.808992 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:16.809475 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:16.811258 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:16.811264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:16.811269 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:16.811542 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:16.811803 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:16.811807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:16.811812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:16.811912 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:16.812256 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:16.812261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:16.812362 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 23:44:16.812475 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 23:44:16.812629 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab.
Entry for principal oryx/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab.
I20260928 23:44:16.832885 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44619
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:16.943938 29693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:16.944218 29693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:16.944295 29693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:16.948630 29693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:16.948721 29693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:16.948779 29693 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:44:16.948817 29693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:16.954300 29693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:16.955462 29693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:16.956607 29693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:16.967367 29699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:16.967372 29701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:16.967439 29698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:16.967600 29693 server_base.cc:1034] running on GCE node
I20260928 23:44:16.968281 29693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:16.969033 29693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:16.970230 29693 hybrid_clock.cc:648] HybridClock initialized: now 1790639056970201 us; error 49 us; skew 500 ppm
Sep 28 23:44:16 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:16.973289 29693 init.cc:377] Logged in from keytab as oryx/127.21.112.66@KRBTEST.COM (short username oryx)
I20260928 23:44:16.974381 29693 webserver.cc:460] Webserver started at http://127.21.112.66:44411/ using document root <none> and password file <none>
I20260928 23:44:16.975000 29693 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:16.975093 29693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:16.975340 29693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:16.977607 29693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "cbfd0adc9f45be3c9457e6189e3d92e4"
server_key_iv: "d2066247e5bfed6a5a526acc10d7224a"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.978230 29693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "cbfd0adc9f45be3c9457e6189e3d92e4"
server_key_iv: "d2066247e5bfed6a5a526acc10d7224a"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.982295 29693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:16.984990 29709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:16.986366 29693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:16.986511 29693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
format_stamp: "Formatted at 2026-09-28 23:44:16 on dist-test-slave-c3s8"
server_key: "cbfd0adc9f45be3c9457e6189e3d92e4"
server_key_iv: "d2066247e5bfed6a5a526acc10d7224a"
server_key_version: "encryptionkey@0"
I20260928 23:44:16.986631 29693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:17.007156 29693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:17.010469 29693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:17.010684 29693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:17.012269 29693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:17.012341 29693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:17.012418 29693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:17.012460 29693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:17.023821 29693 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:45293
I20260928 23:44:17.023998 29825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:45293 every 8 connection(s)
I20260928 23:44:17.024852 29693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:17.026994 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.013360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:50689 (local address 127.21.112.126:44619)
0928 23:44:17.013506 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.013510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.014754 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 23:44:17.015880 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.015884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.015946 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.016051 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.017470 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.018062 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.019437 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.019751 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.019873 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.023108 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:17.023113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:17.023115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:17.023159 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:17.024787 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.025307 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.025311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.025313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.025402 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.025833 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.025837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.025839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.026041 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:17.026158 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.026670 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.026814 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:44:17.027802 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.013990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.21.112.66:50689)
0928 23:44:17.014574 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.014619 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.015668 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.016226 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.016233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.016719 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:17.017313 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.017334 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.018197 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.018206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.019308 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.019319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.019919 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.019923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.020102 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.020790 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:17.020818 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:17.022913 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:17.024918 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.024924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.024939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.025201 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.025506 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.025509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.025512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.025732 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.026158 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:17.026165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:17.026416 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.026610 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.026977 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":147,"threads_started":1}
I20260928 23:44:17.029026 29826 heartbeater.cc:347] Connected to a master server at 127.21.112.126:44619
I20260928 23:44:17.029256 29826 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:17.029574 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 29693
I20260928 23:44:17.029696 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:17.029942 29826 heartbeater.cc:511] Master 127.21.112.126:44619 requested a full tablet report, sending...
I20260928 23:44:17.030030 21953 external_mini_cluster.cc:1428] Setting key e1d720f6b56f9416be7dcc32b417b8ce
I20260928 23:44:17.031004 29478 ts_manager.cc:194] Registered new tserver with Master: 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.031862 29478 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:50689
I20260928 23:44:17.039649 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.031458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.21.112.66:59055)
0928 23:44:17.031731 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.031746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.031841 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.032104 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.032110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.032349 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:17.032509 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.032521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.033663 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.033665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.034229 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.034236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.034846 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.034849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.034933 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.035477 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:17.035489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:17.035792 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:17.037776 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.037779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.037781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.038004 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.038466 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.038472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.038477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.038588 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.039217 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:17.039220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:17.039303 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.039390 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.039506 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260928 23:44:17.039803 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.031427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:59055 (local address 127.21.112.126:44619)
0928 23:44:17.031557 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.031561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.031826 (+ 265us) server_negotiation.cc:407] Connection header received
0928 23:44:17.031922 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.031927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.031971 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.032056 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.032647 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.033475 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.034339 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.034718 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.034823 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.035921 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:17.035928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:17.035934 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:17.035964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:17.037636 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.038199 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.038205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.038209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.038310 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.038706 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.038711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.038715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.039099 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:17.039210 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.039484 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.039614 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for oryx/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab.
Entry for principal oryx/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab.
I20260928 23:44:17.065703 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44619
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:17.186594 29835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:17.186896 29835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:17.186947 29835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:17.191179 29835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:17.191244 29835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:17.191285 29835 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:44:17.191319 29835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:17.197578 29835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.112.126:44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:17.198691 29835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:17.199710 29835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:17.210026 29840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:17.210055 29835 server_base.cc:1034] running on GCE node
W20260928 23:44:17.210011 29841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:17.210006 29843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:17.210661 29835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:17.211279 29835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:17.212445 29835 hybrid_clock.cc:648] HybridClock initialized: now 1790639057212430 us; error 35 us; skew 500 ppm
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639057, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:17.215301 29835 init.cc:377] Logged in from keytab as oryx/127.21.112.67@KRBTEST.COM (short username oryx)
I20260928 23:44:17.216223 29835 webserver.cc:460] Webserver started at http://127.21.112.67:42219/ using document root <none> and password file <none>
I20260928 23:44:17.216776 29835 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:17.216835 29835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:17.217067 29835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:17.219133 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "c5cfa2da891949669be99443bf5785c6"
format_stamp: "Formatted at 2026-09-28 23:44:17 on dist-test-slave-c3s8"
server_key: "2282f84016f98da9d77ef0db8d2b49e5"
server_key_iv: "90462d0fd62304ee3ad70dbbde197f5b"
server_key_version: "encryptionkey@0"
I20260928 23:44:17.219715 29835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "c5cfa2da891949669be99443bf5785c6"
format_stamp: "Formatted at 2026-09-28 23:44:17 on dist-test-slave-c3s8"
server_key: "2282f84016f98da9d77ef0db8d2b49e5"
server_key_iv: "90462d0fd62304ee3ad70dbbde197f5b"
server_key_version: "encryptionkey@0"
I20260928 23:44:17.223685 29835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:17.226054 29850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:17.227276 29835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:17.227407 29835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "c5cfa2da891949669be99443bf5785c6"
format_stamp: "Formatted at 2026-09-28 23:44:17 on dist-test-slave-c3s8"
server_key: "2282f84016f98da9d77ef0db8d2b49e5"
server_key_iv: "90462d0fd62304ee3ad70dbbde197f5b"
server_key_version: "encryptionkey@0"
I20260928 23:44:17.227520 29835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:17.265089 29835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:17.268705 29835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:17.268930 29835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:17.270567 29835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:17.270641 29835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:17.270707 29835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:17.270753 29835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639057, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:17.281944 29835 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:45189
I20260928 23:44:17.281967 29966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:45189 every 8 connection(s)
I20260928 23:44:17.283218 29835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:17.286309 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.271612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:45139 (local address 127.21.112.126:44619)
0928 23:44:17.271775 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.271781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.272722 (+ 941us) server_negotiation.cc:407] Connection header received
0928 23:44:17.274171 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.274175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.274239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.274372 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.276084 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.276860 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.278361 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.278612 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.278738 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.281865 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:17.281868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:17.281870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:17.281896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:17.283917 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.284606 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.284609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.284610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.284685 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.285036 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.285039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.285041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.285243 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:17.285346 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.286021 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.286150 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:44:17.287742 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.272068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.21.112.67:45139)
0928 23:44:17.272561 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.272616 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.273928 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.274596 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.274605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.275118 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:17.275899 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.275927 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.277018 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.277022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.278210 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.278221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.278793 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.278798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.279008 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.279765 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:17.279785 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:17.281578 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:17.284050 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.284056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.284072 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.284465 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.284788 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.284791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.284793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.284934 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.285355 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:17.285363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:17.285712 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.285948 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.286516 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260928 23:44:17.289364 29967 heartbeater.cc:347] Connected to a master server at 127.21.112.126:44619
I20260928 23:44:17.289557 29967 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:17.290207 29967 heartbeater.cc:511] Master 127.21.112.126:44619 requested a full tablet report, sending...
I20260928 23:44:17.291390 29479 ts_manager.cc:194] Registered new tserver with Master: c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189)
I20260928 23:44:17.292183 29479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:45139
I20260928 23:44:17.292898 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 29835
I20260928 23:44:17.293092 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:17.293444 21953 external_mini_cluster.cc:1428] Setting key 08a8d26a3cd3a783fd54daf1a70163cf
I20260928 23:44:17.295594 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:44:17.301303 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.291982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.21.112.67:33939)
0928 23:44:17.292233 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.292248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.292361 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.293848 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.293852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.294089 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:17.294222 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.294230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.295228 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.295232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.296001 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.296011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.296627 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.296629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.296719 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.297346 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:17.297358 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:17.297774 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:17.299577 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.299583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.299588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.299909 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.300301 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.300306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.300310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.300409 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.300792 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:17.300797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:17.300909 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.301024 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.301149 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260928 23:44:17.301450 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.291902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:33939 (local address 127.21.112.126:44619)
0928 23:44:17.292032 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.292036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.292334 (+ 298us) server_negotiation.cc:407] Connection header received
0928 23:44:17.292449 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.292451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.292519 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.292605 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.294362 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.295108 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.296118 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.296511 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.296646 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.297904 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:17.297907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:17.297910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:17.297934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:17.299468 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.300103 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.300108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.300110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.300200 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.300505 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.300507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.300508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.300696 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:17.300772 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.301116 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.301244 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:44:17.320964 29529 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:17.314716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.21.112.126:44619)
0928 23:44:17.314878 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.314882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.314897 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:17.314940 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.314942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.314983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.315070 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.315785 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.316390 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.317252 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.317468 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.317615 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.320813 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35012: BlockingRecv error: recv got EOF from 127.0.0.1:35012 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:17.330447 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.322553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.21.112.126:44619)
0928 23:44:17.322701 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.322708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.322886 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:44:17.323004 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.323008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.323058 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.323141 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.323767 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.324266 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.325039 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.325270 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.325384 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.327675 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:17.327679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:17.327681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:17.327704 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:17.328913 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.329287 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.329289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.329290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.329359 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.329652 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.329669 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.329671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.329844 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:17.329946 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.330142 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.330275 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:44:17.333644 29479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:17.336673 29479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:17.351450 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.345580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.21.112.65:35211)
0928 23:44:17.346461 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.346471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.346497 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:44:17.346754 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.346760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.346980 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.347132 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.347478 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.348639 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.350178 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.350757 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.350846 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.351018 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.351193 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.351296 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":110,"threads_started":1}
I20260928 23:44:17.351459 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.345359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:35211 (local address 127.0.0.1:45584)
0928 23:44:17.345886 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.345938 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.346605 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.347129 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.347133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.347153 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.347347 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.347358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.348753 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.348756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.350040 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.350050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.350887 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.350893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.350997 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.351033 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.351149 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.351296 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":82,"thread_start_us":99,"threads_started":1}
I20260928 23:44:17.351933 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.346303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.21.112.66:45293)
0928 23:44:17.347293 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.347305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.347326 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:44:17.347403 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.347407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.347607 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.347764 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.347973 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.349008 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.350259 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.350792 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.350920 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.351078 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.351711 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.351817 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260928 23:44:17.353080 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.346277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.21.112.67:45189)
0928 23:44:17.346859 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.346870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.346889 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:44:17.346952 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.346958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.347112 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.347261 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.347520 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.348409 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.351079 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.352041 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.352649 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.352802 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.352850 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.352931 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":89,"threads_started":1}
I20260928 23:44:17.354497 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 6d93ea447da149c083b11b2a568d6b31 (DEFAULT_TABLE table=test-table [id=09c33852dbc64c948ee845a523624562]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:17.351464 29992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.346121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45293 (local address 127.0.0.1:51962)
0928 23:44:17.346615 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.346629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.346725 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.347756 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.347759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.347777 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.347858 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.347864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.349133 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.349137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.350135 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.350148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.350946 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.350950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.351040 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.351070 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.351147 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.351296 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260928 23:44:17.355399 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.345571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45189 (local address 127.0.0.1:48710)
0928 23:44:17.346072 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.346085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.346178 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.347265 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.347267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.347281 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.347425 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.347434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.348516 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.348522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.350039 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.350050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.352316 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.352320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.352421 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.352451 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.352506 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.352571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":68,"thread_start_us":93,"threads_started":1}
I20260928 23:44:17.355605 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d93ea447da149c083b11b2a568d6b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.357831 29759 tablet_service.cc:1467] Processing CreateTablet for tablet 6d93ea447da149c083b11b2a568d6b31 (DEFAULT_TABLE table=test-table [id=09c33852dbc64c948ee845a523624562]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:17.358712 29759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d93ea447da149c083b11b2a568d6b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.358487 29900 tablet_service.cc:1467] Processing CreateTablet for tablet 6d93ea447da149c083b11b2a568d6b31 (DEFAULT_TABLE table=test-table [id=09c33852dbc64c948ee845a523624562]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:17.359452 29900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d93ea447da149c083b11b2a568d6b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.363234 29995 tablet_bootstrap.cc:492] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Bootstrap starting.
I20260928 23:44:17.365701 29995 tablet_bootstrap.cc:654] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.365736 29996 tablet_bootstrap.cc:492] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Bootstrap starting.
I20260928 23:44:17.366762 29995 log.cc:824] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:17.367452 29997 tablet_bootstrap.cc:492] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Bootstrap starting.
I20260928 23:44:17.367959 29996 tablet_bootstrap.cc:654] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.368713 29995 tablet_bootstrap.cc:492] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: No bootstrap required, opened a new log
I20260928 23:44:17.368919 29995 ts_tablet_manager.cc:1397] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:44:17.368893 29996 log.cc:824] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:17.369628 29997 tablet_bootstrap.cc:654] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.370464 29997 log.cc:824] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:17.371026 29996 tablet_bootstrap.cc:492] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: No bootstrap required, opened a new log
I20260928 23:44:17.371253 29996 ts_tablet_manager.cc:1397] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 23:44:17.372463 29997 tablet_bootstrap.cc:492] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: No bootstrap required, opened a new log
I20260928 23:44:17.372643 29997 ts_tablet_manager.cc:1397] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 23:44:17.372882 29995 raft_consensus.cc:348] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.373137 29995 raft_consensus.cc:374] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.373188 29995 raft_consensus.cc:729] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.373696 29995 consensus_queue.cc:260] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.374666 29995 ts_tablet_manager.cc:1428] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:44:17.375042 29683 heartbeater.cc:503] Master 127.21.112.126:44619 was elected leader, sending a full tablet report...
I20260928 23:44:17.375377 29996 raft_consensus.cc:348] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.375630 29996 raft_consensus.cc:374] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.375706 29996 raft_consensus.cc:729] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e99c9e7394d66a1dc1fc31308cc15, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.376324 29996 consensus_queue.cc:260] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.376950 29826 heartbeater.cc:503] Master 127.21.112.126:44619 was elected leader, sending a full tablet report...
I20260928 23:44:17.377328 29996 ts_tablet_manager.cc:1428] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 23:44:17.377216 29997 raft_consensus.cc:348] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.377462 29997 raft_consensus.cc:374] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.377516 29997 raft_consensus.cc:729] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.377947 29997 consensus_queue.cc:260] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.378762 29997 ts_tablet_manager.cc:1428] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:44:17.378835 29967 heartbeater.cc:503] Master 127.21.112.126:44619 was elected leader, sending a full tablet report...
W20260928 23:44:17.526475 29827 tablet.cc:2369] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:44:17.536938 29968 tablet.cc:2369] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:44:17.550724 29684 tablet.cc:2369] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:44:17.656772 30001 raft_consensus.cc:482] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:17.657013 30001 raft_consensus.cc:504] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.658259 30001 leader_election.cc:288] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.663240 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.658632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45189 (local address 127.21.112.65:49781)
0928 23:44:17.658946 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.658959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.659102 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.659533 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.659536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.659555 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.659665 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.659671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.660664 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.660689 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.661817 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.661828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.662741 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.662745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.662851 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.662878 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.662971 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.663043 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":144,"threads_started":1}
I20260928 23:44:17.663272 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.658718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:49781 (local address 127.21.112.67:45189)
0928 23:44:17.658884 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.658892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.659058 (+ 166us) server_negotiation.cc:407] Connection header received
0928 23:44:17.659346 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.659349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.659407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.659515 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.659799 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.660557 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.661941 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.662607 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.662703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.662743 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.663001 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.663141 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:44:17.663964 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.659288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:35277 (local address 127.21.112.66:45293)
0928 23:44:17.659399 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.659402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.659882 (+ 480us) server_negotiation.cc:407] Connection header received
0928 23:44:17.659982 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.659984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.660024 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.660080 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.660396 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.661064 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.662537 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.663167 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.663297 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.663352 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.663792 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.663863 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:44:17.663964 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.659341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45293 (local address 127.21.112.65:35277)
0928 23:44:17.659783 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.659798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.659899 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.660115 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.660118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.660135 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.660250 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.660256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.661267 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.661270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.662374 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.662386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.663297 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.663302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.663627 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.663673 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.663767 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.663833 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":98,"threads_started":1}
I20260928 23:44:17.663993 29920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d93ea447da149c083b11b2a568d6b31" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cfa2da891949669be99443bf5785c6" is_pre_election: true
I20260928 23:44:17.664376 29920 raft_consensus.cc:2454] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 0.
I20260928 23:44:17.664710 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d93ea447da149c083b11b2a568d6b31" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" is_pre_election: true
I20260928 23:44:17.664949 29570 leader_election.cc:302] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8600defd184efeb8444be2038f0ac2, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:17.664994 29779 raft_consensus.cc:2454] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 0.
I20260928 23:44:17.665266 30001 raft_consensus.cc:2781] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:17.665364 30001 raft_consensus.cc:482] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:17.665431 30001 raft_consensus.cc:3037] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.666534 30001 raft_consensus.cc:504] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.666908 30001 leader_election.cc:288] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 election: Requested vote from peers c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.667284 29920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d93ea447da149c083b11b2a568d6b31" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cfa2da891949669be99443bf5785c6"
I20260928 23:44:17.667311 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d93ea447da149c083b11b2a568d6b31" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
I20260928 23:44:17.667421 29779 raft_consensus.cc:3037] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.667423 29920 raft_consensus.cc:3037] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.668869 29779 raft_consensus.cc:2454] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 1.
I20260928 23:44:17.669279 29920 raft_consensus.cc:2454] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 1.
I20260928 23:44:17.669339 29570 leader_election.cc:302] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e99c9e7394d66a1dc1fc31308cc15, bd8600defd184efeb8444be2038f0ac2; no voters:
I20260928 23:44:17.669588 30001 raft_consensus.cc:2781] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:17.669852 30001 raft_consensus.cc:686] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 1 LEADER]: Becoming Leader. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Running, Role: LEADER
I20260928 23:44:17.670233 30001 consensus_queue.cc:237] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } }
I20260928 23:44:17.674082 29479 catalog_manager.cc:5495] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd8600defd184efeb8444be2038f0ac2 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "bd8600defd184efeb8444be2038f0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } health_report { overall_health: UNKNOWN } } }
I20260928 23:44:17.716806 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.711986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.21.112.65:35211)
0928 23:44:17.712150 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.712157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.712177 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:44:17.712336 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.712342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.712435 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.712559 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:17.712793 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.713770 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.714721 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.715184 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.715307 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.715567 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:17.716297 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:17.716432 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.716509 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.716589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:44:17.724815 29920 raft_consensus.cc:1261] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Refusing update from remote peer bd8600defd184efeb8444be2038f0ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:17.724874 29779 raft_consensus.cc:1261] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Refusing update from remote peer bd8600defd184efeb8444be2038f0ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:17.725657 30001 consensus_queue.cc:1035] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:17.725901 30006 consensus_queue.cc:1035] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:17.735289 30010 mvcc.cc:205] Tried to move back new op lower bound from 7334457580432838656 to 7334457580220366848. Current Snapshot: MvccSnapshot[applied={T|T < 7334457580432838656}]
I20260928 23:44:17.744315 29479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:17.744518 29479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:17.745401 30011 mvcc.cc:205] Tried to move back new op lower bound from 7334457580432838656 to 7334457580220366848. Current Snapshot: MvccSnapshot[applied={T|T < 7334457580432838656}]
I20260928 23:44:17.748093 29479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Sending DeleteTablet for 3 replicas of tablet 6d93ea447da149c083b11b2a568d6b31
I20260928 23:44:17.750061 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d93ea447da149c083b11b2a568d6b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:17 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:45584
I20260928 23:44:17.751842 29759 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d93ea447da149c083b11b2a568d6b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:17 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51962
I20260928 23:44:17.751847 29900 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d93ea447da149c083b11b2a568d6b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:17 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:48710
I20260928 23:44:17.752291 29758 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d93ea447da149c083b11b2a568d6b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:17 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51962
I20260928 23:44:17.752434 30019 tablet_replica.cc:330] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: stopping tablet replica
I20260928 23:44:17.752897 30019 raft_consensus.cc:2229] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:17.753367 30019 raft_consensus.cc:2258] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:17.754269 29477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:17.754793 29477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:17.755475 30019 ts_tablet_manager.cc:1905] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:17.759111 30018 tablet_replica.cc:330] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: stopping tablet replica
I20260928 23:44:17.759687 30019 ts_tablet_manager.cc:1918] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:17.759907 30019 log.cc:1196] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/6d93ea447da149c083b11b2a568d6b31
I20260928 23:44:17.760341 30019 ts_tablet_manager.cc:1939] T 6d93ea447da149c083b11b2a568d6b31 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting consensus metadata
I20260928 23:44:17.762137 29465 catalog_manager.cc:4841] TS 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293): tablet 6d93ea447da149c083b11b2a568d6b31 (table test-table [id=09c33852dbc64c948ee845a523624562]) successfully deleted
W20260928 23:44:17.762562 29465 catalog_manager.cc:4808] TS 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293): delete failed for tablet 6d93ea447da149c083b11b2a568d6b31 because the tablet was not found. No further retry: Not found: Tablet not found: 6d93ea447da149c083b11b2a568d6b31
I20260928 23:44:17.763585 29900 tablet_service.cc:1467] Processing CreateTablet for tablet c2303df77e554985b2d0cf4dae4eb28c (DEFAULT_TABLE table=test-table [id=07f9c8fe6e864e69a11160a164dd0d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:17.763731 29758 tablet_service.cc:1467] Processing CreateTablet for tablet c2303df77e554985b2d0cf4dae4eb28c (DEFAULT_TABLE table=test-table [id=07f9c8fe6e864e69a11160a164dd0d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:17.763826 29617 tablet_service.cc:1467] Processing CreateTablet for tablet c2303df77e554985b2d0cf4dae4eb28c (DEFAULT_TABLE table=test-table [id=07f9c8fe6e864e69a11160a164dd0d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:17.763952 29900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2303df77e554985b2d0cf4dae4eb28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.763996 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2303df77e554985b2d0cf4dae4eb28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.764165 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2303df77e554985b2d0cf4dae4eb28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.766846 29997 tablet_bootstrap.cc:492] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Bootstrap starting.
I20260928 23:44:17.767426 30020 tablet_replica.cc:330] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: stopping tablet replica
I20260928 23:44:17.767839 30020 raft_consensus.cc:2229] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:17.767964 29997 tablet_bootstrap.cc:654] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.768281 30020 raft_consensus.cc:2258] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:17.768663 29995 tablet_bootstrap.cc:492] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Bootstrap starting.
I20260928 23:44:17.769161 29996 tablet_bootstrap.cc:492] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Bootstrap starting.
I20260928 23:44:17.769773 29995 tablet_bootstrap.cc:654] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.770231 29996 tablet_bootstrap.cc:654] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.770776 29997 tablet_bootstrap.cc:492] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: No bootstrap required, opened a new log
I20260928 23:44:17.770864 29997 ts_tablet_manager.cc:1397] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 23:44:17.771409 29997 raft_consensus.cc:348] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.771533 29997 raft_consensus.cc:374] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.771567 29997 raft_consensus.cc:729] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.772051 29995 tablet_bootstrap.cc:492] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: No bootstrap required, opened a new log
I20260928 23:44:17.772164 29995 ts_tablet_manager.cc:1397] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:44:17.772563 29995 raft_consensus.cc:348] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.772693 29995 raft_consensus.cc:374] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.772735 29995 raft_consensus.cc:729] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.772864 30020 ts_tablet_manager.cc:1905] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:17.772881 29995 consensus_queue.cc:260] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.773612 29995 ts_tablet_manager.cc:1428] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:17.773787 29997 consensus_queue.cc:260] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.774155 29997 ts_tablet_manager.cc:1428] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:44:17.779536 30020 ts_tablet_manager.cc:1918] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:17.779655 30020 log.cc:1196] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/6d93ea447da149c083b11b2a568d6b31
I20260928 23:44:17.780055 30020 ts_tablet_manager.cc:1939] T 6d93ea447da149c083b11b2a568d6b31 P c5cfa2da891949669be99443bf5785c6: Deleting consensus metadata
I20260928 23:44:17.781482 29465 catalog_manager.cc:4841] TS c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189): tablet 6d93ea447da149c083b11b2a568d6b31 (table test-table [id=09c33852dbc64c948ee845a523624562]) successfully deleted
I20260928 23:44:17.781621 30018 raft_consensus.cc:2229] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:17.782028 30018 raft_consensus.cc:2258] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:17.783592 30018 ts_tablet_manager.cc:1905] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:17.786496 29996 tablet_bootstrap.cc:492] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: No bootstrap required, opened a new log
I20260928 23:44:17.786620 29996 ts_tablet_manager.cc:1397] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260928 23:44:17.786834 30018 ts_tablet_manager.cc:1918] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:17.786932 30018 log.cc:1196] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/6d93ea447da149c083b11b2a568d6b31
I20260928 23:44:17.787266 30018 ts_tablet_manager.cc:1939] T 6d93ea447da149c083b11b2a568d6b31 P bd8600defd184efeb8444be2038f0ac2: Deleting consensus metadata
I20260928 23:44:17.787261 29996 raft_consensus.cc:348] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.787397 29996 raft_consensus.cc:374] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.787439 29996 raft_consensus.cc:729] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e99c9e7394d66a1dc1fc31308cc15, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.787611 29996 consensus_queue.cc:260] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.787995 29996 ts_tablet_manager.cc:1428] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:17.788594 29463 catalog_manager.cc:4841] TS bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211): tablet 6d93ea447da149c083b11b2a568d6b31 (table test-table [id=09c33852dbc64c948ee845a523624562]) successfully deleted
I20260928 23:44:17.884373 30003 raft_consensus.cc:482] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:17.884562 30003 raft_consensus.cc:504] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.886145 30003 leader_election.cc:288] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.892453 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.886793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43233 (local address 127.21.112.65:35211)
0928 23:44:17.886928 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.886933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.887249 (+ 316us) server_negotiation.cc:407] Connection header received
0928 23:44:17.887483 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.887487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.887542 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.887594 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.888039 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.888823 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.890908 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.891555 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.891655 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.891731 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.892227 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.892314 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:17.893146 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.886390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:35211 (local address 127.21.112.67:43233)
0928 23:44:17.887150 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.887166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.887340 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.887761 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.887764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.887783 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.887902 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.887909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.888978 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.888981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.890757 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.890769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.891883 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.891886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.892059 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.892085 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.892873 (+ 788us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.892966 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260928 23:44:17.893903 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.886893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:35055 (local address 127.21.112.66:45293)
0928 23:44:17.887037 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.887045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.887307 (+ 262us) server_negotiation.cc:407] Connection header received
0928 23:44:17.887519 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.887526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.887586 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.887642 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.888126 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.889064 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.892205 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.893007 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.893120 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.893191 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.893617 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.893733 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:17.893935 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2303df77e554985b2d0cf4dae4eb28c" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2" is_pre_election: true
I20260928 23:44:17.894253 29637 raft_consensus.cc:2454] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 0.
I20260928 23:44:17.894413 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.886500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45293 (local address 127.21.112.67:35055)
0928 23:44:17.887221 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.887238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.887389 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.887817 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.887822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.887843 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.887988 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.887997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.889225 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.889230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.892035 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.892050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.893295 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.893300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.893434 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.893469 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.894110 (+ 641us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.894214 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":45,"thread_start_us":78,"threads_started":1}
I20260928 23:44:17.894762 29851 leader_election.cc:302] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8600defd184efeb8444be2038f0ac2, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:17.894948 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2303df77e554985b2d0cf4dae4eb28c" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" is_pre_election: true
I20260928 23:44:17.895068 30003 raft_consensus.cc:2781] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:17.895125 30003 raft_consensus.cc:482] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:17.895112 29779 raft_consensus.cc:2454] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 0.
I20260928 23:44:17.895165 30003 raft_consensus.cc:3037] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.896533 30003 raft_consensus.cc:504] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.896975 30003 leader_election.cc:288] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 election: Requested vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.897480 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2303df77e554985b2d0cf4dae4eb28c" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
I20260928 23:44:17.897617 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2303df77e554985b2d0cf4dae4eb28c" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2"
I20260928 23:44:17.897637 29779 raft_consensus.cc:3037] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.897768 29637 raft_consensus.cc:3037] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.898947 29779 raft_consensus.cc:2454] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 1.
I20260928 23:44:17.899273 29637 raft_consensus.cc:2454] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 1.
I20260928 23:44:17.899323 29853 leader_election.cc:302] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e99c9e7394d66a1dc1fc31308cc15, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:17.899497 30003 raft_consensus.cc:2781] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:17.899662 30003 raft_consensus.cc:686] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 1 LEADER]: Becoming Leader. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Running, Role: LEADER
I20260928 23:44:17.899950 30003 consensus_queue.cc:237] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.902647 29478 catalog_manager.cc:5495] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cfa2da891949669be99443bf5785c6 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } health_report { overall_health: HEALTHY } } }
Sep 28 23:44:17 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639056, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.126@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:17.952081 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.943324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44619 (local address 127.0.0.1:35026)
0928 23:44:17.943764 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.943781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.943993 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.944273 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.944276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.944712 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:17.944806 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.944812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.945587 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.945593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.946162 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.946169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.946620 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.946623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.946693 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.947254 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:17.947266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:17.948654 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:17.950690 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.950711 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.950716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.950946 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.951193 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:17.951195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:17.951197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:17.951276 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:17.951612 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:17.951617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:17.951731 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.951821 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.951933 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":148,"threads_started":1}
I20260928 23:44:17.952103 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.943448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.21.112.126:44619)
0928 23:44:17.943753 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.943763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.943938 (+ 175us) server_negotiation.cc:407] Connection header received
0928 23:44:17.944115 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.944120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.944164 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.944256 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:17.944925 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.945458 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.946277 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.946510 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.946620 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.948816 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:17.948820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:17.948822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:17.948845 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:17.950583 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.951035 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.951038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.951039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.951114 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:17.951352 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:17.951354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:17.951355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:17.951527 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:17.951601 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.951886 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.951984 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":90,"threads_started":1}
I20260928 23:44:17.956382 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:35026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:44:17.957142 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:17.962095 29758 tablet_service.cc:1467] Processing CreateTablet for tablet d099a8f8a00f40e387190f4f747893d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5564258d4c404810b06f0e510c995a6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:44:17.962136 29900 tablet_service.cc:1467] Processing CreateTablet for tablet d099a8f8a00f40e387190f4f747893d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5564258d4c404810b06f0e510c995a6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:44:17.962092 29617 tablet_service.cc:1467] Processing CreateTablet for tablet d099a8f8a00f40e387190f4f747893d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5564258d4c404810b06f0e510c995a6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:44:17.962476 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d099a8f8a00f40e387190f4f747893d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.962488 29900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d099a8f8a00f40e387190f4f747893d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.962488 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d099a8f8a00f40e387190f4f747893d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:17.965183 29996 tablet_bootstrap.cc:492] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: Bootstrap starting.
I20260928 23:44:17.965250 29997 tablet_bootstrap.cc:492] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: Bootstrap starting.
I20260928 23:44:17.966224 29996 tablet_bootstrap.cc:654] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.966259 29997 tablet_bootstrap.cc:654] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.966920 29995 tablet_bootstrap.cc:492] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: Bootstrap starting.
I20260928 23:44:17.967396 29997 tablet_bootstrap.cc:492] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: No bootstrap required, opened a new log
I20260928 23:44:17.967468 29997 ts_tablet_manager.cc:1397] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:17.967777 29995 tablet_bootstrap.cc:654] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:17.967944 29997 raft_consensus.cc:348] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.968065 29997 raft_consensus.cc:374] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.968096 29997 raft_consensus.cc:729] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.968231 29997 consensus_queue.cc:260] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.968514 29997 ts_tablet_manager.cc:1428] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:17.968698 29996 tablet_bootstrap.cc:492] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: No bootstrap required, opened a new log
I20260928 23:44:17.968775 29996 ts_tablet_manager.cc:1397] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:44:17.968722 30003 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:17.968847 30003 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:17.969242 29996 raft_consensus.cc:348] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.969338 29996 raft_consensus.cc:374] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.969360 29996 raft_consensus.cc:729] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e99c9e7394d66a1dc1fc31308cc15, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.969477 29996 consensus_queue.cc:260] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.969604 30002 raft_consensus.cc:482] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:17.969709 30002 raft_consensus.cc:504] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.970139 29996 ts_tablet_manager.cc:1428] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 23:44:17.971071 30002 leader_election.cc:288] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189)
I20260928 23:44:17.971107 29995 tablet_bootstrap.cc:492] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: No bootstrap required, opened a new log
I20260928 23:44:17.971181 29995 ts_tablet_manager.cc:1397] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 23:44:17.971609 29995 raft_consensus.cc:348] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.971752 29995 raft_consensus.cc:374] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:17.971796 29995 raft_consensus.cc:729] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Initialized, Role: FOLLOWER
I20260928 23:44:17.971926 29995 consensus_queue.cc:260] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.972136 30038 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:17.972222 29995 ts_tablet_manager.cc:1428] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:17.972275 30038 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:17.972398 30006 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:17.972535 30006 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:17.973881 30003 raft_consensus.cc:482] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:17.973973 30003 raft_consensus.cc:504] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.974340 30003 leader_election.cc:288] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.974596 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d099a8f8a00f40e387190f4f747893d7" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2" is_pre_election: true
I20260928 23:44:17.974668 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d099a8f8a00f40e387190f4f747893d7" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" is_pre_election: true
I20260928 23:44:17.974713 29637 raft_consensus.cc:2454] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 0.
I20260928 23:44:17.974781 29779 raft_consensus.cc:2454] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 0.
I20260928 23:44:17.975018 29851 leader_election.cc:302] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8600defd184efeb8444be2038f0ac2, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:17.975246 30003 raft_consensus.cc:2781] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:17.975306 30003 raft_consensus.cc:482] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:17.975358 30003 raft_consensus.cc:3037] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.976513 30003 raft_consensus.cc:504] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.976867 30003 leader_election.cc:288] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 election: Requested vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:17.977104 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.971833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:38343 (local address 127.21.112.65:35211)
0928 23:44:17.971986 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.971992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.972009 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:17.972073 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.972076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.972119 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.972198 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.973243 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.973959 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.975901 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.976543 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.976662 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.976716 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.976890 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.976985 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:44:17.977133 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d099a8f8a00f40e387190f4f747893d7" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2"
I20260928 23:44:17.977150 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d099a8f8a00f40e387190f4f747893d7" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
I20260928 23:44:17.977245 29779 raft_consensus.cc:3037] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.977248 29637 raft_consensus.cc:3037] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:17.977342 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.971266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:35211 (local address 127.21.112.66:38343)
0928 23:44:17.971527 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.971543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.971644 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.972886 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.972891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.972916 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.973085 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.973095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.974110 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.974113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.975778 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.975788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.976646 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.976651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.976757 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.976779 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.977097 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.977180 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":121,"threads_started":1}
I20260928 23:44:17.978333 29637 raft_consensus.cc:2454] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 1.
I20260928 23:44:17.978332 29779 raft_consensus.cc:2454] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 1.
I20260928 23:44:17.978598 29853 leader_election.cc:302] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e99c9e7394d66a1dc1fc31308cc15, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:17.978754 30003 raft_consensus.cc:2781] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:17.978806 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d099a8f8a00f40e387190f4f747893d7" candidate_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2" is_pre_election: true
I20260928 23:44:17.978845 30003 raft_consensus.cc:686] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [term 1 LEADER]: Becoming Leader. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Running, Role: LEADER
I20260928 23:44:17.978997 29637 raft_consensus.cc:2379] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4b5e99c9e7394d66a1dc1fc31308cc15 in current term 1: Already voted for candidate c5cfa2da891949669be99443bf5785c6 in this term.
I20260928 23:44:17.978972 30003 consensus_queue.cc:237] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:17.979725 30003 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: TxnStatusTablet state changed. Reason: New leader c5cfa2da891949669be99443bf5785c6. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:17.979818 30003 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:44:17.980623 29478 catalog_manager.cc:5495] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cfa2da891949669be99443bf5785c6 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } health_report { overall_health: HEALTHY } } }
I20260928 23:44:17.981088 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.971335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45189 (local address 127.21.112.66:57493)
0928 23:44:17.976030 (+ 4695us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.976044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.976179 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.976399 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.976402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.976417 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:17.976533 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.976538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.978215 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.978220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.979470 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.979480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.980685 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.980691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.980796 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.980831 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.980909 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.980973 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4624,"thread_start_us":82,"threads_started":1}
I20260928 23:44:17.981128 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.971625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:57493 (local address 127.21.112.67:45189)
0928 23:44:17.971735 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.971742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.976141 (+ 4399us) server_negotiation.cc:407] Connection header received
0928 23:44:17.976259 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.976262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.976301 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.976391 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:17.977632 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.978103 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.979586 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.980166 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.980276 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.980334 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.980948 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.981025 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:44:17.981451 29920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d099a8f8a00f40e387190f4f747893d7" candidate_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cfa2da891949669be99443bf5785c6" is_pre_election: true
I20260928 23:44:17.981715 30042 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:44:17.981876 30042 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:44:17.981899 29712 leader_election.cc:302] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e99c9e7394d66a1dc1fc31308cc15; no voters: bd8600defd184efeb8444be2038f0ac2, c5cfa2da891949669be99443bf5785c6
I20260928 23:44:17.982769 30038 raft_consensus.cc:2726] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 23:44:17.995106 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.990881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.21.112.67:45189)
0928 23:44:17.991015 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:17.991020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:17.991086 (+ 66us) server_negotiation.cc:407] Connection header received
0928 23:44:17.991310 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:17.991313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:17.991381 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:17.991486 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:17.991863 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.992374 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.993678 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.993893 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.993978 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.994314 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:17.994748 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:17.994839 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:17.994942 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:44:17.994994 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:44:17.995167 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:17.990728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45189 (local address 127.0.0.1:48722)
0928 23:44:17.991008 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:17.991022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:17.991169 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:17.991620 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:17.991624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:17.991632 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:44:17.991737 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.991743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.992488 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.992492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.993528 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:17.993567 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:17.994006 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:17.994010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:17.994104 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:17.994159 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:44:17.994846 (+ 687us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:44:17.994861 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 23:44:17.994935 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:44:17.995025 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 23:44:17.998117 29779 raft_consensus.cc:1261] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Refusing update from remote peer c5cfa2da891949669be99443bf5785c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:17.998646 30003 consensus_queue.cc:1035] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:17.998749 29637 raft_consensus.cc:1261] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Refusing update from remote peer c5cfa2da891949669be99443bf5785c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:17.999167 30003 consensus_queue.cc:1035] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.000279 30038 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: TxnStatusTablet state changed. Reason: New leader c5cfa2da891949669be99443bf5785c6. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:18.000399 30038 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:18.001012 30006 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: TxnStatusTablet state changed. Reason: New leader c5cfa2da891949669be99443bf5785c6. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:18.001114 30006 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:18.002640 30003 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:18.002746 30003 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:44:18.002857 30003 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:18.002942 30003 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P c5cfa2da891949669be99443bf5785c6: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:44:18.002971 30038 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:18.003068 30038 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P 4b5e99c9e7394d66a1dc1fc31308cc15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:18.003316 30009 mvcc.cc:205] Tried to move back new op lower bound from 7334457581554880512 to 7334457581483368448. Current Snapshot: MvccSnapshot[applied={T|T < 7334457581554880512}]
I20260928 23:44:18.003614 30006 tablet_replica.cc:439] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } } }
I20260928 23:44:18.003721 30006 tablet_replica.cc:442] T d099a8f8a00f40e387190f4f747893d7 P bd8600defd184efeb8444be2038f0ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:44:18.012559 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.008109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.21.112.67:45189)
0928 23:44:18.008220 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:18.008225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:18.008238 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:44:18.008279 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:18.008282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:18.008339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:18.008420 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:18.008642 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.009419 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.010983 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.011357 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.011459 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.011912 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:18.012021 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:18.012108 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:18.012280 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:44:18.012368 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:44:18.026612 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.022169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:38425 (local address 127.21.112.67:45189)
0928 23:44:18.022305 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:18.022310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:18.022655 (+ 345us) server_negotiation.cc:407] Connection header received
0928 23:44:18.022844 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:18.022847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:18.022898 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:18.022983 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:18.023272 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.024072 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.025348 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.025717 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.025812 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.026134 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:18.026228 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:18.026298 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:18.026411 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 23:44:18.026463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:44:18.026654 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.022182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45189 (local address 127.21.112.67:38425)
0928 23:44:18.022561 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:18.022581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:18.022731 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:18.022998 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:18.023015 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:18.023027 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:44:18.023178 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:18.023183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.024212 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.024215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:18.025225 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:18.025235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.025841 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.025845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:18.025946 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.026002 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:44:18.026316 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:44:18.026323 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:44:18.026391 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:44:18.026476 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":1,"thread_start_us":166,"threads_started":1}
I20260928 23:44:18.033879 29779 raft_consensus.cc:1261] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Refusing update from remote peer c5cfa2da891949669be99443bf5785c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:18.033996 29637 raft_consensus.cc:1261] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Refusing update from remote peer c5cfa2da891949669be99443bf5785c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:18.034394 30047 consensus_queue.cc:1035] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.034641 30047 consensus_queue.cc:1035] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.077380 29478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:18.077591 29478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:18.079706 29478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Sending DeleteTablet for 3 replicas of tablet c2303df77e554985b2d0cf4dae4eb28c
I20260928 23:44:18.080420 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet c2303df77e554985b2d0cf4dae4eb28c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:45584
I20260928 23:44:18.080451 29899 tablet_service.cc:1514] Processing DeleteTablet for tablet c2303df77e554985b2d0cf4dae4eb28c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:48710
I20260928 23:44:18.080533 29758 tablet_service.cc:1514] Processing DeleteTablet for tablet c2303df77e554985b2d0cf4dae4eb28c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51962
I20260928 23:44:18.080699 30019 tablet_replica.cc:330] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: stopping tablet replica
I20260928 23:44:18.080698 30020 tablet_replica.cc:330] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: stopping tablet replica
I20260928 23:44:18.080811 30020 raft_consensus.cc:2229] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:18.080811 30019 raft_consensus.cc:2229] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:18.080892 30018 tablet_replica.cc:330] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: stopping tablet replica
I20260928 23:44:18.080935 30019 raft_consensus.cc:2258] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.081032 30018 raft_consensus.cc:2229] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:18.081130 30020 raft_consensus.cc:2258] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.081151 30018 raft_consensus.cc:2258] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.081904 30020 ts_tablet_manager.cc:1905] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.082075 30018 ts_tablet_manager.cc:1905] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.085249 30018 ts_tablet_manager.cc:1918] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:44:18.085448 30020 ts_tablet_manager.cc:1918] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:44:18.085508 30018 log.cc:1196] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/c2303df77e554985b2d0cf4dae4eb28c
I20260928 23:44:18.085589 30020 log.cc:1196] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/c2303df77e554985b2d0cf4dae4eb28c
I20260928 23:44:18.085880 30020 ts_tablet_manager.cc:1939] T c2303df77e554985b2d0cf4dae4eb28c P c5cfa2da891949669be99443bf5785c6: Deleting consensus metadata
I20260928 23:44:18.085912 30018 ts_tablet_manager.cc:1939] T c2303df77e554985b2d0cf4dae4eb28c P bd8600defd184efeb8444be2038f0ac2: Deleting consensus metadata
I20260928 23:44:18.086529 30019 ts_tablet_manager.cc:1905] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.086738 29465 catalog_manager.cc:4841] TS c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189): tablet c2303df77e554985b2d0cf4dae4eb28c (table test-table [id=07f9c8fe6e864e69a11160a164dd0d14]) successfully deleted
I20260928 23:44:18.086959 29463 catalog_manager.cc:4841] TS bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211): tablet c2303df77e554985b2d0cf4dae4eb28c (table test-table [id=07f9c8fe6e864e69a11160a164dd0d14]) successfully deleted
I20260928 23:44:18.088913 30019 ts_tablet_manager.cc:1918] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:44:18.089004 30019 log.cc:1196] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/c2303df77e554985b2d0cf4dae4eb28c
I20260928 23:44:18.089262 30019 ts_tablet_manager.cc:1939] T c2303df77e554985b2d0cf4dae4eb28c P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting consensus metadata
I20260928 23:44:18.090292 29465 catalog_manager.cc:4841] TS 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293): tablet c2303df77e554985b2d0cf4dae4eb28c (table test-table [id=07f9c8fe6e864e69a11160a164dd0d14]) successfully deleted
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[29434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:44:18.115859 30034 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:18.108121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.21.112.126:44619)
0928 23:44:18.108270 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:18.108277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:18.108472 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:44:18.108719 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:18.108722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:18.108774 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:18.108875 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:18.109744 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.110335 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.111286 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.111569 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.111703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.115708 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35028: BlockingRecv error: recv got EOF from 127.0.0.1:35028 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[29434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM
I20260928 23:44:18.127753 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.117837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.21.112.126:44619)
0928 23:44:18.118002 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:18.118007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:18.118078 (+ 71us) server_negotiation.cc:407] Connection header received
0928 23:44:18.118208 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:18.118212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:18.118266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:18.118375 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:18.118991 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.119913 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.120784 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.121178 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.121300 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.124230 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:18.124233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:18.124235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:18.124260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:18.125955 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:18.126491 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:18.126494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:18.126495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:18.126570 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:18.126909 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:18.126911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:18.126912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:18.127089 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:18.127197 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:18.127489 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 23:44:18.127594 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:44:18.130841 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:18.131311 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:18.136368 29617 tablet_service.cc:1467] Processing CreateTablet for tablet e0d8388001b54b7896ec7c273ab61fb4 (DEFAULT_TABLE table=test-table [id=10138cc1e8874c10938f7fbae177a15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:18.136409 29899 tablet_service.cc:1467] Processing CreateTablet for tablet e0d8388001b54b7896ec7c273ab61fb4 (DEFAULT_TABLE table=test-table [id=10138cc1e8874c10938f7fbae177a15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:18.136742 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d8388001b54b7896ec7c273ab61fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:18.136790 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d8388001b54b7896ec7c273ab61fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:18.138911 29997 tablet_bootstrap.cc:492] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Bootstrap starting.
I20260928 23:44:18.139176 29758 tablet_service.cc:1467] Processing CreateTablet for tablet e0d8388001b54b7896ec7c273ab61fb4 (DEFAULT_TABLE table=test-table [id=10138cc1e8874c10938f7fbae177a15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:18.139386 29995 tablet_bootstrap.cc:492] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Bootstrap starting.
I20260928 23:44:18.139454 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d8388001b54b7896ec7c273ab61fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:18.139851 29997 tablet_bootstrap.cc:654] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:18.140208 29995 tablet_bootstrap.cc:654] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:18.141455 29995 tablet_bootstrap.cc:492] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: No bootstrap required, opened a new log
I20260928 23:44:18.141589 29995 ts_tablet_manager.cc:1397] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:18.142043 29995 raft_consensus.cc:348] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.142175 29995 raft_consensus.cc:374] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:18.142212 29995 raft_consensus.cc:729] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Initialized, Role: FOLLOWER
I20260928 23:44:18.142344 29995 consensus_queue.cc:260] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.142639 29995 ts_tablet_manager.cc:1428] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:18.142849 29996 tablet_bootstrap.cc:492] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Bootstrap starting.
I20260928 23:44:18.143460 29997 tablet_bootstrap.cc:492] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: No bootstrap required, opened a new log
I20260928 23:44:18.143573 29997 ts_tablet_manager.cc:1397] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 23:44:18.143739 29996 tablet_bootstrap.cc:654] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:18.144083 29997 raft_consensus.cc:348] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.144204 29997 raft_consensus.cc:374] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:18.144245 29997 raft_consensus.cc:729] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Initialized, Role: FOLLOWER
I20260928 23:44:18.144390 29997 consensus_queue.cc:260] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.144654 29997 ts_tablet_manager.cc:1428] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:18.146914 29996 tablet_bootstrap.cc:492] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: No bootstrap required, opened a new log
I20260928 23:44:18.147017 29996 ts_tablet_manager.cc:1397] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 23:44:18.147527 29996 raft_consensus.cc:348] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.147663 29996 raft_consensus.cc:374] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:18.147712 29996 raft_consensus.cc:729] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e99c9e7394d66a1dc1fc31308cc15, State: Initialized, Role: FOLLOWER
I20260928 23:44:18.147868 29996 consensus_queue.cc:260] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.148576 29996 ts_tablet_manager.cc:1428] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:18.213584 30006 raft_consensus.cc:482] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:18.213726 30006 raft_consensus.cc:504] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.214187 30006 leader_election.cc:288] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293), c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189)
I20260928 23:44:18.214699 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d8388001b54b7896ec7c273ab61fb4" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" is_pre_election: true
I20260928 23:44:18.214704 29920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d8388001b54b7896ec7c273ab61fb4" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cfa2da891949669be99443bf5785c6" is_pre_election: true
I20260928 23:44:18.214859 29779 raft_consensus.cc:2454] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 0.
I20260928 23:44:18.214861 29920 raft_consensus.cc:2454] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 0.
I20260928 23:44:18.215171 29570 leader_election.cc:302] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e99c9e7394d66a1dc1fc31308cc15, bd8600defd184efeb8444be2038f0ac2; no voters:
I20260928 23:44:18.215427 30006 raft_consensus.cc:2781] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:18.215503 30006 raft_consensus.cc:482] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:18.215553 30006 raft_consensus.cc:3037] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:18.216971 30006 raft_consensus.cc:504] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.217387 30006 leader_election.cc:288] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 election: Requested vote from peers 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293), c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189)
I20260928 23:44:18.217808 29920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d8388001b54b7896ec7c273ab61fb4" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cfa2da891949669be99443bf5785c6"
I20260928 23:44:18.217810 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d8388001b54b7896ec7c273ab61fb4" candidate_uuid: "bd8600defd184efeb8444be2038f0ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
I20260928 23:44:18.217937 29779 raft_consensus.cc:3037] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:18.217941 29920 raft_consensus.cc:3037] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:18.218858 29920 raft_consensus.cc:2454] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 1.
I20260928 23:44:18.219089 29779 raft_consensus.cc:2454] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8600defd184efeb8444be2038f0ac2 in term 1.
I20260928 23:44:18.219219 29570 leader_election.cc:302] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8600defd184efeb8444be2038f0ac2, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:18.219391 30006 raft_consensus.cc:2781] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:18.219542 30006 raft_consensus.cc:686] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 1 LEADER]: Becoming Leader. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Running, Role: LEADER
I20260928 23:44:18.219681 30006 consensus_queue.cc:237] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.221529 29478 catalog_manager.cc:5495] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd8600defd184efeb8444be2038f0ac2 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "bd8600defd184efeb8444be2038f0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20260928 23:44:18.248921 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.245414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.21.112.65:35211)
0928 23:44:18.245576 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:18.245583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:18.245600 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:18.245708 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:18.245710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:18.245754 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:18.245819 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:18.246100 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.246606 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.247815 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.248044 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.248125 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.248403 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:18.248473 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:18.248529 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:18.248645 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 23:44:18.248703 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:44:18.251091 29779 raft_consensus.cc:1261] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Refusing update from remote peer bd8600defd184efeb8444be2038f0ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:18.251134 29920 raft_consensus.cc:1261] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Refusing update from remote peer bd8600defd184efeb8444be2038f0ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:18.251552 30006 consensus_queue.cc:1035] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.251775 30001 consensus_queue.cc:1035] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.260634 29478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:18.260775 29478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:18.262775 29478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Sending DeleteTablet for 3 replicas of tablet e0d8388001b54b7896ec7c273ab61fb4
I20260928 23:44:18.263357 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d8388001b54b7896ec7c273ab61fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:45584
I20260928 23:44:18.263543 30018 tablet_replica.cc:330] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: stopping tablet replica
I20260928 23:44:18.263528 29899 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d8388001b54b7896ec7c273ab61fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:48710
I20260928 23:44:18.263549 29758 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d8388001b54b7896ec7c273ab61fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51962
I20260928 23:44:18.263660 30018 raft_consensus.cc:2229] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:18.263697 30020 tablet_replica.cc:330] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: stopping tablet replica
I20260928 23:44:18.263794 30020 raft_consensus.cc:2229] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:18.263850 30018 raft_consensus.cc:2258] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.263911 30020 raft_consensus.cc:2258] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.264056 30019 tablet_replica.cc:330] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: stopping tablet replica
I20260928 23:44:18.264190 30019 raft_consensus.cc:2229] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:18.264328 30019 raft_consensus.cc:2258] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.264657 30020 ts_tablet_manager.cc:1905] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.264959 30019 ts_tablet_manager.cc:1905] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.265736 30018 ts_tablet_manager.cc:1905] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.267185 30020 ts_tablet_manager.cc:1918] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:18.267264 30020 log.cc:1196] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/e0d8388001b54b7896ec7c273ab61fb4
I20260928 23:44:18.267441 30019 ts_tablet_manager.cc:1918] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:18.267514 30020 ts_tablet_manager.cc:1939] T e0d8388001b54b7896ec7c273ab61fb4 P c5cfa2da891949669be99443bf5785c6: Deleting consensus metadata
I20260928 23:44:18.267524 30019 log.cc:1196] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/e0d8388001b54b7896ec7c273ab61fb4
I20260928 23:44:18.267804 30019 ts_tablet_manager.cc:1939] T e0d8388001b54b7896ec7c273ab61fb4 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting consensus metadata
I20260928 23:44:18.268183 30018 ts_tablet_manager.cc:1918] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:18.268239 30018 log.cc:1196] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/e0d8388001b54b7896ec7c273ab61fb4
I20260928 23:44:18.268472 30018 ts_tablet_manager.cc:1939] T e0d8388001b54b7896ec7c273ab61fb4 P bd8600defd184efeb8444be2038f0ac2: Deleting consensus metadata
I20260928 23:44:18.268616 29465 catalog_manager.cc:4841] TS 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293): tablet e0d8388001b54b7896ec7c273ab61fb4 (table test-table [id=10138cc1e8874c10938f7fbae177a15e]) successfully deleted
I20260928 23:44:18.269132 29463 catalog_manager.cc:4841] TS bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211): tablet e0d8388001b54b7896ec7c273ab61fb4 (table test-table [id=10138cc1e8874c10938f7fbae177a15e]) successfully deleted
I20260928 23:44:18.269198 29465 catalog_manager.cc:4841] TS c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189): tablet e0d8388001b54b7896ec7c273ab61fb4 (table test-table [id=10138cc1e8874c10938f7fbae177a15e]) successfully deleted
I20260928 23:44:18.269249 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:18.269728 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:18.274720 29617 tablet_service.cc:1467] Processing CreateTablet for tablet 45f9ce2ee943458f976146403b242604 (DEFAULT_TABLE table=test-table [id=be080b127da04ebf8778827e6d39bb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:18.274778 29758 tablet_service.cc:1467] Processing CreateTablet for tablet 45f9ce2ee943458f976146403b242604 (DEFAULT_TABLE table=test-table [id=be080b127da04ebf8778827e6d39bb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:18.275031 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f9ce2ee943458f976146403b242604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:18.275030 29758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f9ce2ee943458f976146403b242604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:18.277442 29995 tablet_bootstrap.cc:492] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2: Bootstrap starting.
I20260928 23:44:18.277487 29996 tablet_bootstrap.cc:492] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Bootstrap starting.
I20260928 23:44:18.278309 29995 tablet_bootstrap.cc:654] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:18.278375 29996 tablet_bootstrap.cc:654] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:18.279086 29899 tablet_service.cc:1467] Processing CreateTablet for tablet 45f9ce2ee943458f976146403b242604 (DEFAULT_TABLE table=test-table [id=be080b127da04ebf8778827e6d39bb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:18.279307 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f9ce2ee943458f976146403b242604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:18.279412 29995 tablet_bootstrap.cc:492] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2: No bootstrap required, opened a new log
I20260928 23:44:18.279484 29995 ts_tablet_manager.cc:1397] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:18.279850 29995 raft_consensus.cc:348] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.279953 29995 raft_consensus.cc:374] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:18.279981 29995 raft_consensus.cc:729] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8600defd184efeb8444be2038f0ac2, State: Initialized, Role: FOLLOWER
I20260928 23:44:18.280112 29995 consensus_queue.cc:260] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.280350 29995 ts_tablet_manager.cc:1428] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:44:18.281426 29997 tablet_bootstrap.cc:492] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Bootstrap starting.
I20260928 23:44:18.282271 29997 tablet_bootstrap.cc:654] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:18.282562 29996 tablet_bootstrap.cc:492] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: No bootstrap required, opened a new log
I20260928 23:44:18.282661 29996 ts_tablet_manager.cc:1397] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 23:44:18.283182 29996 raft_consensus.cc:348] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.283303 29996 raft_consensus.cc:374] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:18.283337 29996 raft_consensus.cc:729] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e99c9e7394d66a1dc1fc31308cc15, State: Initialized, Role: FOLLOWER
I20260928 23:44:18.283406 29997 tablet_bootstrap.cc:492] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: No bootstrap required, opened a new log
I20260928 23:44:18.283473 29997 ts_tablet_manager.cc:1397] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:18.283460 29996 consensus_queue.cc:260] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.283738 29996 ts_tablet_manager.cc:1428] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:18.283897 29997 raft_consensus.cc:348] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.283991 29997 raft_consensus.cc:374] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:18.284021 29997 raft_consensus.cc:729] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Initialized, Role: FOLLOWER
I20260928 23:44:18.284137 29997 consensus_queue.cc:260] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.284431 29997 ts_tablet_manager.cc:1428] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:44:18.309607 30026 raft_consensus.cc:482] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:18.309754 30026 raft_consensus.cc:504] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.310207 30026 leader_election.cc:288] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:18.310740 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f9ce2ee943458f976146403b242604" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2" is_pre_election: true
I20260928 23:44:18.310776 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f9ce2ee943458f976146403b242604" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" is_pre_election: true
I20260928 23:44:18.310921 29779 raft_consensus.cc:2454] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 0.
I20260928 23:44:18.310926 29637 raft_consensus.cc:2454] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 0.
I20260928 23:44:18.311271 29853 leader_election.cc:302] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e99c9e7394d66a1dc1fc31308cc15, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:18.311482 30026 raft_consensus.cc:2781] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:18.311558 30026 raft_consensus.cc:482] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:18.311604 30026 raft_consensus.cc:3037] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:18.312741 30026 raft_consensus.cc:504] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.313215 30026 leader_election.cc:288] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 election: Requested vote from peers bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211), 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293)
I20260928 23:44:18.313596 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f9ce2ee943458f976146403b242604" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8600defd184efeb8444be2038f0ac2"
I20260928 23:44:18.313705 29779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f9ce2ee943458f976146403b242604" candidate_uuid: "c5cfa2da891949669be99443bf5785c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15"
I20260928 23:44:18.313762 29637 raft_consensus.cc:3037] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:18.313838 29779 raft_consensus.cc:3037] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:18.314765 29637 raft_consensus.cc:2454] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 1.
I20260928 23:44:18.315034 29779 raft_consensus.cc:2454] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5cfa2da891949669be99443bf5785c6 in term 1.
I20260928 23:44:18.315137 29851 leader_election.cc:302] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd8600defd184efeb8444be2038f0ac2, c5cfa2da891949669be99443bf5785c6; no voters:
I20260928 23:44:18.315506 30026 raft_consensus.cc:2781] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:18.315614 30026 raft_consensus.cc:686] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 1 LEADER]: Becoming Leader. State: Replica: c5cfa2da891949669be99443bf5785c6, State: Running, Role: LEADER
I20260928 23:44:18.315761 30026 consensus_queue.cc:237] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } }
I20260928 23:44:18.317636 29478 catalog_manager.cc:5495] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5cfa2da891949669be99443bf5785c6 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "c5cfa2da891949669be99443bf5785c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5cfa2da891949669be99443bf5785c6" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45189 } health_report { overall_health: HEALTHY } } }
I20260928 23:44:18.337208 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.333866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.21.112.67:45189)
0928 23:44:18.333993 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:18.333998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:18.334012 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:18.334132 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:18.334135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:18.334186 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:18.334265 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:18.334521 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.335034 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.335946 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:18.336300 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:18.336439 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:18.336729 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:18.336812 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:18.336900 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:18.337005 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:44:18.337055 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:44:18.343851 29779 raft_consensus.cc:1261] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Refusing update from remote peer c5cfa2da891949669be99443bf5785c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:18.343947 29637 raft_consensus.cc:1261] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Refusing update from remote peer c5cfa2da891949669be99443bf5785c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:18.344246 30026 consensus_queue.cc:1035] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e99c9e7394d66a1dc1fc31308cc15" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.344434 30047 consensus_queue.cc:1035] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8600defd184efeb8444be2038f0ac2" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:18.378965 29478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:18.379135 29478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:18.381394 29478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6396f2b38db4df187754ef81d9dc3cf: Sending DeleteTablet for 3 replicas of tablet 45f9ce2ee943458f976146403b242604
I20260928 23:44:18.382123 29617 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f9ce2ee943458f976146403b242604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:45584
I20260928 23:44:18.382349 30018 tablet_replica.cc:330] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2: stopping tablet replica
I20260928 23:44:18.382315 29899 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f9ce2ee943458f976146403b242604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:48710
I20260928 23:44:18.382411 29758 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f9ce2ee943458f976146403b242604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:18 UTC) from {username='oryx', principal='oryx/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51962
I20260928 23:44:18.382473 30018 raft_consensus.cc:2229] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:18.382524 30020 tablet_replica.cc:330] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: stopping tablet replica
I20260928 23:44:18.382576 30019 tablet_replica.cc:330] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: stopping tablet replica
I20260928 23:44:18.382598 30018 raft_consensus.cc:2258] T 45f9ce2ee943458f976146403b242604 P bd8600defd184efeb8444be2038f0ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.382644 30020 raft_consensus.cc:2229] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:18.382689 30019 raft_consensus.cc:2229] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:18.382802 30019 raft_consensus.cc:2258] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.382860 30020 raft_consensus.cc:2258] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:18.383247 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 29552
I20260928 23:44:18.383646 30020 ts_tablet_manager.cc:1905] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.383630 30019 ts_tablet_manager.cc:1905] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:18.387195 30020 ts_tablet_manager.cc:1918] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:44:18.387276 30019 ts_tablet_manager.cc:1918] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:44:18.387318 30020 log.cc:1196] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/45f9ce2ee943458f976146403b242604
I20260928 23:44:18.387368 30019 log.cc:1196] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/45f9ce2ee943458f976146403b242604
I20260928 23:44:18.387574 30020 ts_tablet_manager.cc:1939] T 45f9ce2ee943458f976146403b242604 P c5cfa2da891949669be99443bf5785c6: Deleting consensus metadata
I20260928 23:44:18.387640 30019 ts_tablet_manager.cc:1939] T 45f9ce2ee943458f976146403b242604 P 4b5e99c9e7394d66a1dc1fc31308cc15: Deleting consensus metadata
I20260928 23:44:18.388370 29465 catalog_manager.cc:4841] TS c5cfa2da891949669be99443bf5785c6 (127.21.112.67:45189): tablet 45f9ce2ee943458f976146403b242604 (table test-table [id=be080b127da04ebf8778827e6d39bb2e]) successfully deleted
I20260928 23:44:18.388577 29465 catalog_manager.cc:4841] TS 4b5e99c9e7394d66a1dc1fc31308cc15 (127.21.112.66:45293): tablet 45f9ce2ee943458f976146403b242604 (table test-table [id=be080b127da04ebf8778827e6d39bb2e]) successfully deleted
W20260928 23:44:18.392665 29463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:35211 (error 108)
I20260928 23:44:18.393514 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 29693
I20260928 23:44:18.395516 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:18.393574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:35211 (local address 127.0.0.1:45616)
0928 23:44:18.395288 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:18.395381 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:35211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1630,"thread_start_us":1553,"threads_started":1}
W20260928 23:44:18.395720 29463 catalog_manager.cc:4584] TS bd8600defd184efeb8444be2038f0ac2 (127.21.112.65:35211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45f9ce2ee943458f976146403b242604: Network error: Client connection negotiation failed: client connection to 127.21.112.65:35211: connect: Connection refused (error 111)
I20260928 23:44:18.400983 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 29835
I20260928 23:44:18.408133 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 29447
2026-09-28T23:44:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3287 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[30084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[30084](info): set up 2 sockets
Sep 28 23:44:18 dist-test-slave-c3s8 krb5kdc[30084](info): commencing operation
krb5kdc: starting...
W20260928 23:44:19.502751 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:19 dist-test-slave-c3s8 krb5kdc[30084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:44:19.629828 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:38277
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:19.742991 30097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:19.743325 30097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:19.743400 30097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:19.747586 30097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:19.747685 30097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:19.747720 30097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:19.747761 30097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:19.747792 30097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:19.753348 30097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:38277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:19.754518 30097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:19.755592 30097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:19.764933 30105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:19.764878 30102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:19.764878 30103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:19.764973 30097 server_base.cc:1034] running on GCE node
I20260928 23:44:19.766024 30097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:19.767247 30097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:19.768479 30097 hybrid_clock.cc:648] HybridClock initialized: now 1790639059768439 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T23:44:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1368 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:19 dist-test-slave-c3s8 krb5kdc[30110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:19 dist-test-slave-c3s8 krb5kdc[30110](info): set up 2 sockets
Sep 28 23:44:19 dist-test-slave-c3s8 krb5kdc[30110](info): commencing operation
krb5kdc: starting...
W20260928 23:44:20.874855 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:20 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:44:21.004925 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:33855
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:21.117951 30123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:21.118221 30123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:21.118261 30123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:21.122295 30123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:21.122350 30123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:21.122370 30123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:21.122387 30123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:21.122404 30123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:21.127658 30123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:21.128652 30123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:21.129637 30123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:21.138154 30128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:21.138154 30131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:21.138173 30129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:21.138684 30123 server_base.cc:1034] running on GCE node
I20260928 23:44:21.139216 30123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:21.140267 30123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:21.141430 30123 hybrid_clock.cc:648] HybridClock initialized: now 1790639061141403 us; error 38 us; skew 500 ppm
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:21.144627 30123 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:44:21.145673 30123 webserver.cc:460] Webserver started at http://127.21.112.126:45551/ using document root <none> and password file <none>
I20260928 23:44:21.146260 30123 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:21.146351 30123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:21.146571 30123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:21.148617 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "2492f0e47a7d45c4bb9386ce402aa08f"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "d15eae0147cc25a413a836af6a9536cb"
server_key_iv: "bc8027c01faed976b1013d12162ed08e"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.149183 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "2492f0e47a7d45c4bb9386ce402aa08f"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "d15eae0147cc25a413a836af6a9536cb"
server_key_iv: "bc8027c01faed976b1013d12162ed08e"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.152937 30123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 23:44:21.155948 30138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.157429 30123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:21.157629 30123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "2492f0e47a7d45c4bb9386ce402aa08f"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "d15eae0147cc25a413a836af6a9536cb"
server_key_iv: "bc8027c01faed976b1013d12162ed08e"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.157765 30123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:21.189589 30123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:21.193137 30123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:21.193421 30123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:21.200654 30123 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:33855
I20260928 23:44:21.200664 30190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:33855 every 8 connection(s)
I20260928 23:44:21.201650 30123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:21.201967 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30123
I20260928 23:44:21.202122 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:21.202482 21953 external_mini_cluster.cc:1428] Setting key fb74842b6de60f8e39821c8540bf1ce1
I20260928 23:44:21.204855 30191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:21.210484 30191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Bootstrap starting.
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:21.212818 30191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:21.213676 30191 log.cc:824] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:21.215965 30191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: No bootstrap required, opened a new log
I20260928 23:44:21.216822 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.204038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.21.112.126:33855)
0928 23:44:21.204483 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:21.204498 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:21.204535 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:44:21.205339 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:21.205358 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:21.205763 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:21.206122 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:21.206913 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.207961 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.208816 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.209281 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.209449 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.211978 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:21.211983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:21.211996 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:21.212023 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:21.213995 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.214463 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.214471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.214482 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.214589 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.214878 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.214880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.214881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.215328 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:21.215473 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:21.215767 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 23:44:21.216053 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260928 23:44:21.219520 30191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:21.219769 30191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:21.219851 30191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2492f0e47a7d45c4bb9386ce402aa08f, State: Initialized, Role: FOLLOWER
I20260928 23:44:21.220263 30191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:21.220409 30191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:21.220494 30191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:21.220613 30191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:21.221482 30191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:21.221837 30191 leader_election.cc:302] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2492f0e47a7d45c4bb9386ce402aa08f; no voters:
I20260928 23:44:21.222153 30191 leader_election.cc:288] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:21.222326 30196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:21.222653 30196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 LEADER]: Becoming Leader. State: Replica: 2492f0e47a7d45c4bb9386ce402aa08f, State: Running, Role: LEADER
I20260928 23:44:21.222972 30196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:21.223433 30191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:21.225205 30198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2492f0e47a7d45c4bb9386ce402aa08f. Latest consensus state: current_term: 1 leader_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } } }
I20260928 23:44:21.225353 30198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:21.225674 30197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } } }
I20260928 23:44:21.226390 30197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: This master's current role is: LEADER
W20260928 23:44:21.228084 30211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:44:21.228153 30211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:44:21.228235 30210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:21.228929 30210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:21.234663 30210 catalog_manager.cc:1347] Generated new cluster ID: 679693d2bdf640c3bd9c77403a036b7a
I20260928 23:44:21.234728 30210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:21.254359 30210 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:21.255714 30210 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:21.268406 30210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Generated new TSK 0
I20260928 23:44:21.269277 30210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:44:21.313081 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33855
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:21.427887 30217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:21.428158 30217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:21.428233 30217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:21.432366 30217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:21.432453 30217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:21.432546 30217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:21.437980 30217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:21.439186 30217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:21.440253 30217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:21.450166 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:21.450268 30217 server_base.cc:1034] running on GCE node
W20260928 23:44:21.450148 30222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:21.450168 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:21.450834 30217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:21.451493 30217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:21.452714 30217 hybrid_clock.cc:648] HybridClock initialized: now 1790639061452680 us; error 50 us; skew 500 ppm
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:21.455752 30217 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:44:21.456601 30217 webserver.cc:460] Webserver started at http://127.21.112.65:42361/ using document root <none> and password file <none>
I20260928 23:44:21.457124 30217 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:21.457181 30217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:21.457343 30217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:21.459455 30217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "9ad94f1900d0494daab7c5cfc16ead6b"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "ffbdd19802cb7c0d2493d0f44a0cbc22"
server_key_iv: "db0466d58a8bb05f4c5a2482dc837504"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.459977 30217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "9ad94f1900d0494daab7c5cfc16ead6b"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "ffbdd19802cb7c0d2493d0f44a0cbc22"
server_key_iv: "db0466d58a8bb05f4c5a2482dc837504"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.464013 30217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 23:44:21.466666 30232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.468063 30217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:21.468182 30217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "9ad94f1900d0494daab7c5cfc16ead6b"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "ffbdd19802cb7c0d2493d0f44a0cbc22"
server_key_iv: "db0466d58a8bb05f4c5a2482dc837504"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.468284 30217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:21.493225 30217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:21.496667 30217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:21.496841 30217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:21.497413 30217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:21.498368 30217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:21.498420 30217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.498463 30217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:21.498488 30217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.508348 30217 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:39801
I20260928 23:44:21.508370 30346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:39801 every 8 connection(s)
I20260928 23:44:21.509428 30217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:21.510162 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30217
I20260928 23:44:21.510329 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:21.510669 21953 external_mini_cluster.cc:1428] Setting key d597fbb228e156270eb9fade60269608
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:44:21.527002 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.511607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:34373 (local address 127.21.112.126:33855)
0928 23:44:21.511765 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:21.511770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:21.513238 (+ 1468us) server_negotiation.cc:407] Connection header received
0928 23:44:21.514683 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:21.514686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:21.514736 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:21.514815 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:21.516213 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.516779 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.518878 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.519212 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.519311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.522562 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:21.522571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:21.522576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:21.522614 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:21.524333 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.524932 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.524935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.524939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.525028 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.525577 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.525583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.525588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.525861 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:21.525969 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:21.526582 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 23:44:21.526743 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:44:21.527925 30349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.512192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:34373)
0928 23:44:21.513069 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:21.513133 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:21.514398 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:21.515043 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:21.515054 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:21.515501 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:21.516040 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:21.516054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.516931 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.516939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:21.518715 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:21.518733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.519402 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.519409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:21.519683 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.520462 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:21.520485 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:21.522368 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:21.524440 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:21.524445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:21.524459 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:21.524812 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:21.525140 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:21.525142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:21.525144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:21.525406 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:21.525991 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:21.525997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:21.526288 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 23:44:21.526450 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:44:21.526728 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":159,"threads_started":1}
I20260928 23:44:21.529426 30347 heartbeater.cc:347] Connected to a master server at 127.21.112.126:33855
I20260928 23:44:21.529711 30347 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:21.530243 30347 heartbeater.cc:511] Master 127.21.112.126:33855 requested a full tablet report, sending...
I20260928 23:44:21.531867 30154 ts_manager.cc:194] Registered new tserver with Master: 9ad94f1900d0494daab7c5cfc16ead6b (127.21.112.65:39801)
I20260928 23:44:21.533056 30154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:34373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:44:21.545238 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33855
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:21.661621 30352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:21.661904 30352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:21.661949 30352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:21.666353 30352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:21.666419 30352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:21.666482 30352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:21.672000 30352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:21.673270 30352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:21.674361 30352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:21.684698 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:21.684731 30360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:21.684731 30358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:21.685034 30352 server_base.cc:1034] running on GCE node
I20260928 23:44:21.685470 30352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:21.686229 30352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:21.687420 30352 hybrid_clock.cc:648] HybridClock initialized: now 1790639061687384 us; error 46 us; skew 500 ppm
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:21.690377 30352 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:44:21.691253 30352 webserver.cc:460] Webserver started at http://127.21.112.66:43837/ using document root <none> and password file <none>
I20260928 23:44:21.691844 30352 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:21.691936 30352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:21.692178 30352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:21.694415 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "55ef31aa75a44bc79daf51219273fe95"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "c217b90255babed339fa048272e85ded"
server_key_iv: "ec681fe4caee8a43a30b084b1853f69b"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.695091 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "55ef31aa75a44bc79daf51219273fe95"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "c217b90255babed339fa048272e85ded"
server_key_iv: "ec681fe4caee8a43a30b084b1853f69b"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.698973 30352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:21.701822 30367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.703213 30352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:44:21.703367 30352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "55ef31aa75a44bc79daf51219273fe95"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "c217b90255babed339fa048272e85ded"
server_key_iv: "ec681fe4caee8a43a30b084b1853f69b"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.703490 30352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:21.733916 30352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:21.737663 30352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:21.737881 30352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:21.738533 30352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:21.739454 30352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:21.739531 30352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.739611 30352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:21.739671 30352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.749575 30352 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:45769
I20260928 23:44:21.749596 30481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:45769 every 8 connection(s)
I20260928 23:44:21.750607 30352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:21.752385 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30352
I20260928 23:44:21.752523 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:21.752873 21953 external_mini_cluster.cc:1428] Setting key e83d93287f9094f913d02ea858c277c7
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:44:21.765729 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.752551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:36285 (local address 127.21.112.126:33855)
0928 23:44:21.752730 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:21.752736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:21.753984 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 23:44:21.754901 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:21.754907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:21.754978 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:21.755080 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:21.756335 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.756880 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.758165 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.758399 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.758516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.761082 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:21.761086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:21.761089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:21.761123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:21.763308 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.763912 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.763919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.763922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.764026 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.764438 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.764441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.764442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.764649 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:21.764750 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:21.765370 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 23:44:21.765498 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:44:21.766651 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.753053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:36285)
0928 23:44:21.753813 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:21.753868 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:21.754685 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:21.755285 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:21.755293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:21.755666 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:21.756171 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:21.756184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.757018 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.757021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:21.758033 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:21.758043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.758560 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.758564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:21.758795 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.759408 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:21.759425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:21.760895 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:21.763438 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:21.763444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:21.763458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:21.763801 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:21.764125 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:21.764127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:21.764129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:21.764278 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:21.764790 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:21.764803 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:21.765061 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 23:44:21.765229 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 23:44:21.765591 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:44:21.768110 30482 heartbeater.cc:347] Connected to a master server at 127.21.112.126:33855
I20260928 23:44:21.768410 30482 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:21.768991 30482 heartbeater.cc:511] Master 127.21.112.126:33855 requested a full tablet report, sending...
I20260928 23:44:21.770252 30154 ts_manager.cc:194] Registered new tserver with Master: 55ef31aa75a44bc79daf51219273fe95 (127.21.112.66:45769)
I20260928 23:44:21.770889 30154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:36285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:44:21.786288 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33855
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:21.903991 30487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:21.904268 30487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:21.904314 30487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:21.908788 30487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:21.908855 30487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:21.908946 30487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:21.914573 30487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:21.915717 30487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:21.916805 30487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:21.926923 30495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:21.926990 30493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:21.927060 30487 server_base.cc:1034] running on GCE node
W20260928 23:44:21.926944 30492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:21.927636 30487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:21.928313 30487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:21.929507 30487 hybrid_clock.cc:648] HybridClock initialized: now 1790639061929490 us; error 28 us; skew 500 ppm
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:21.932348 30487 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:44:21.933193 30487 webserver.cc:460] Webserver started at http://127.21.112.67:46113/ using document root <none> and password file <none>
I20260928 23:44:21.933851 30487 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:21.933912 30487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:21.934074 30487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:21.936156 30487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "cbf334823fd84fdabba33774fc732347"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "cc56f9b9dff67655d57bce94052df60b"
server_key_iv: "76f0804591b0a84f064957791a419cd6"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.936659 30487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "cbf334823fd84fdabba33774fc732347"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "cc56f9b9dff67655d57bce94052df60b"
server_key_iv: "76f0804591b0a84f064957791a419cd6"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.940537 30487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:44:21.943382 30502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.944734 30487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:44:21.944870 30487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "cbf334823fd84fdabba33774fc732347"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "cc56f9b9dff67655d57bce94052df60b"
server_key_iv: "76f0804591b0a84f064957791a419cd6"
server_key_version: "encryptionkey@0"
I20260928 23:44:21.944986 30487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:21.967078 30487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:21.970418 30487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:21.970630 30487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:21.971318 30487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:21.972154 30487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:21.972199 30487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.972317 30487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:21.972391 30487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:21.981621 30487 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:34977
I20260928 23:44:21.981640 30615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:34977 every 8 connection(s)
I20260928 23:44:21.982553 30487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:21.983312 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30487
I20260928 23:44:21.983493 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:21.983783 21953 external_mini_cluster.cc:1428] Setting key e67cd393f5dc5c7fff51e4be2f07dc21
Sep 28 23:44:21 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639061, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:21.996170 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.984465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:60311 (local address 127.21.112.126:33855)
0928 23:44:21.984616 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:21.984621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:21.985973 (+ 1352us) server_negotiation.cc:407] Connection header received
0928 23:44:21.986923 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:21.986925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:21.986969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:21.987034 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:21.988127 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.988642 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.989815 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.990028 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.990105 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.992577 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:21.992580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:21.992582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:21.992608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:21.994148 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.994642 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.994645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.994646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.994742 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:21.995094 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:21.995098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:21.995103 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:21.995344 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:21.995427 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:21.995937 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 23:44:21.996043 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:44:21.997184 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:21.985051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:60311)
0928 23:44:21.985829 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:21.985884 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:21.986742 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:21.987166 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:21.987173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:21.987530 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:21.988005 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:21.988018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.988745 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.988747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:21.989713 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:21.989722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:21.990150 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:21.990153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:21.990329 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:21.990871 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:21.990887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:21.992404 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:21.994263 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:21.994268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:21.994282 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:21.994531 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:21.994833 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:21.994835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:21.994836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:21.994972 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:21.995443 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:21.995448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:21.995715 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 23:44:21.995886 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 23:44:21.996141 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":137,"threads_started":1}
I20260928 23:44:21.998445 30616 heartbeater.cc:347] Connected to a master server at 127.21.112.126:33855
I20260928 23:44:21.998658 30616 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:21.999104 30616 heartbeater.cc:511] Master 127.21.112.126:33855 requested a full tablet report, sending...
I20260928 23:44:22.000113 30154 ts_manager.cc:194] Registered new tserver with Master: cbf334823fd84fdabba33774fc732347 (127.21.112.67:34977)
I20260928 23:44:22.000697 30154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:60311
I20260928 23:44:22.010550 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab.
Entry for principal oryx/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab.
WARNING: no policy specified for oryx/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab.
Entry for principal oryx/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab.
WARNING: no policy specified for oryx/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab.
Entry for principal oryx/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab.
I20260928 23:44:22.096549 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30217
I20260928 23:44:22.102643 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30352
I20260928 23:44:22.108330 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30487
I20260928 23:44:22.113739 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30123
I20260928 23:44:22.119937 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:33855
--webserver_interface=127.21.112.126
--webserver_port=45551
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:22.231492 30625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:22.231770 30625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:22.231812 30625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:22.235894 30625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:22.235953 30625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:22.235973 30625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:22.235989 30625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:22.236004 30625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:22.241180 30625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=45551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:22.242278 30625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:22.243276 30625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:22.251902 30633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.251878 30630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.251878 30631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:22.251932 30625 server_base.cc:1034] running on GCE node
I20260928 23:44:22.252720 30625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:22.253955 30625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:22.255146 30625 hybrid_clock.cc:648] HybridClock initialized: now 1790639062255128 us; error 32 us; skew 500 ppm
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:22.258045 30625 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:44:22.259004 30625 webserver.cc:460] Webserver started at http://127.21.112.126:45551/ using document root <none> and password file <none>
I20260928 23:44:22.259533 30625 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:22.259604 30625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:22.263612 30625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:44:22.265821 30640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.267104 30625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:44:22.267199 30625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "2492f0e47a7d45c4bb9386ce402aa08f"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "d15eae0147cc25a413a836af6a9536cb"
server_key_iv: "bc8027c01faed976b1013d12162ed08e"
server_key_version: "encryptionkey@0"
I20260928 23:44:22.267779 30625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:22.282505 30625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:22.285315 30625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:22.285511 30625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:22.292637 30625 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:33855
I20260928 23:44:22.292666 30692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:33855 every 8 connection(s)
I20260928 23:44:22.293782 30625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:22.296382 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30625
I20260928 23:44:22.296959 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:39801
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:33855
--webserver_port=42361
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:44:22.299765 30693 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:44:22.302299 30693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Bootstrap starting.
I20260928 23:44:22.319281 30693 log.cc:824] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:22.325938 30693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:44:22.326330 30693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Bootstrap complete.
I20260928 23:44:22.330569 30693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:22.331002 30693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2492f0e47a7d45c4bb9386ce402aa08f, State: Initialized, Role: FOLLOWER
I20260928 23:44:22.331530 30693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:22.331696 30693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:22.331776 30693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:22.331894 30693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:44:22.333953 30693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:22.334282 30693 leader_election.cc:302] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2492f0e47a7d45c4bb9386ce402aa08f; no voters:
I20260928 23:44:22.334589 30693 leader_election.cc:288] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:44:22.334734 30697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:44:22.334985 30697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [term 2 LEADER]: Becoming Leader. State: Replica: 2492f0e47a7d45c4bb9386ce402aa08f, State: Running, Role: LEADER
I20260928 23:44:22.335292 30697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } }
I20260928 23:44:22.335837 30693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:22.336272 30697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2492f0e47a7d45c4bb9386ce402aa08f. Latest consensus state: current_term: 2 leader_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } } }
I20260928 23:44:22.336386 30697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:22.336776 30701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:22.336808 30698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2492f0e47a7d45c4bb9386ce402aa08f" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33855 } } }
I20260928 23:44:22.337057 30698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:22.340042 30701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:22.340641 30701 catalog_manager.cc:1259] Loaded cluster ID: 679693d2bdf640c3bd9c77403a036b7a
I20260928 23:44:22.340698 30701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:22.343051 30701 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:22.343715 30701 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2492f0e47a7d45c4bb9386ce402aa08f: Loaded TSK: 0
I20260928 23:44:22.345091 30701 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:44:22.415666 30695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:22.415930 30695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:22.415974 30695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:22.420140 30695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:22.420208 30695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:22.420306 30695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:22.425693 30695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.65.keytab
--principal=oryx/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:39801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=42361
--tserver_master_addrs=127.21.112.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:22.426681 30695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:22.427692 30695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:22.437522 30720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.437553 30719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.437553 30722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:22.437698 30695 server_base.cc:1034] running on GCE node
I20260928 23:44:22.438222 30695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:22.438963 30695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:22.440229 30695 hybrid_clock.cc:648] HybridClock initialized: now 1790639062440141 us; error 32 us; skew 500 ppm
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639062, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:22.443112 30695 init.cc:377] Logged in from keytab as oryx/127.21.112.65@KRBTEST.COM (short username oryx)
I20260928 23:44:22.444162 30695 webserver.cc:460] Webserver started at http://127.21.112.65:42361/ using document root <none> and password file <none>
I20260928 23:44:22.444774 30695 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:22.444852 30695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:22.448899 30695 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:22.451156 30729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.452476 30695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:22.452576 30695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "9ad94f1900d0494daab7c5cfc16ead6b"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "ffbdd19802cb7c0d2493d0f44a0cbc22"
server_key_iv: "db0466d58a8bb05f4c5a2482dc837504"
server_key_version: "encryptionkey@0"
I20260928 23:44:22.453163 30695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:22.479698 30695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:22.483238 30695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:22.483458 30695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:22.484097 30695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:22.484961 30695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:22.485008 30695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.485046 30695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:22.485062 30695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.494437 30695 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:39801
I20260928 23:44:22.494459 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:39801 every 8 connection(s)
I20260928 23:44:22.495481 30695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:22.496469 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30695
I20260928 23:44:22.496901 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:45769
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:33855
--webserver_port=43837
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.511049 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.497997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:56787)
0928 23:44:22.498820 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.498878 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.500105 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.501581 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.501594 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.501992 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.502539 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.502554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.504621 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.504624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.505675 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.505685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.506814 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.506817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.506952 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.507584 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.507601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.510208 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":137,"threads_started":1}
W20260928 23:44:22.511287 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.497854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:56787 (local address 127.21.112.126:33855)
0928 23:44:22.498523 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.498544 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.499032 (+ 488us) server_negotiation.cc:407] Connection header received
0928 23:44:22.500583 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.500603 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.501077 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.501473 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.502733 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.504460 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.505800 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.506638 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.506848 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.510458 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:56787: BlockingRecv error: recv got EOF from 127.21.112.65:56787 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":134,"threads_started":1}
W20260928 23:44:22.511777 30843 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:33855 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.112.126:33855: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.517314 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.512210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:58543)
0928 23:44:22.512305 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.512320 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.512409 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.512648 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.512651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.512805 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.512888 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.512893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.513661 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.513666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.514206 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.514214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.514675 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.514679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.514773 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.515262 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.515273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.517114 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:44:22.517350 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.512291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:58543 (local address 127.21.112.126:33855)
0928 23:44:22.512408 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.512413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.512428 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:22.512498 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.512501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.512541 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.512636 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.513007 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.513527 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.514335 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.514565 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.514681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.517252 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:58543: BlockingRecv error: recv got EOF from 127.21.112.65:58543 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.522687 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.517911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:50887)
0928 23:44:22.518006 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.518022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.518095 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.518345 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.518348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.518493 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.518556 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.518560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.519270 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.519276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.519782 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.519789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.520221 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.520224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.520317 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.520795 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.520813 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.522549 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 23:44:22.522908 30843 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:44:22.522948 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.517982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:50887 (local address 127.21.112.126:33855)
0928 23:44:22.518100 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.518104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.518119 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:22.518196 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.518199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.518236 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.518331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.518682 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.519168 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.519894 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.520115 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.520221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.522663 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:50887: BlockingRecv error: recv got EOF from 127.21.112.65:50887 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:44:22.612015 30845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:22.612296 30845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:22.612340 30845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:22.616460 30845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:22.616532 30845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:22.616652 30845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:22.622225 30845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.66.keytab
--principal=oryx/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:45769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=43837
--tserver_master_addrs=127.21.112.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:22.623358 30845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:22.624423 30845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:22.634078 30853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.634061 30852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:22.634438 30845 server_base.cc:1034] running on GCE node
W20260928 23:44:22.634464 30855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:22.634879 30845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:22.635499 30845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:22.636686 30845 hybrid_clock.cc:648] HybridClock initialized: now 1790639062636648 us; error 53 us; skew 500 ppm
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639062, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:22.639730 30845 init.cc:377] Logged in from keytab as oryx/127.21.112.66@KRBTEST.COM (short username oryx)
I20260928 23:44:22.640604 30845 webserver.cc:460] Webserver started at http://127.21.112.66:43837/ using document root <none> and password file <none>
I20260928 23:44:22.641124 30845 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:22.641207 30845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:22.645115 30845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:44:22.647375 30862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.648669 30845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:22.648803 30845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "55ef31aa75a44bc79daf51219273fe95"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "c217b90255babed339fa048272e85ded"
server_key_iv: "ec681fe4caee8a43a30b084b1853f69b"
server_key_version: "encryptionkey@0"
I20260928 23:44:22.649345 30845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:22.664505 30845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:22.667766 30845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:22.667976 30845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:22.668644 30845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:22.669497 30845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:22.669600 30845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.669670 30845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:22.669705 30845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.678732 30845 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:45769
I20260928 23:44:22.678783 30975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:45769 every 8 connection(s)
I20260928 23:44:22.679798 30845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:22.683637 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30845
I20260928 23:44:22.684211 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:34977
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:33855
--webserver_port=46113
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:44:22.692366 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.681745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:33051 (local address 127.21.112.126:33855)
0928 23:44:22.681902 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.681907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.682964 (+ 1057us) server_negotiation.cc:407] Connection header received
0928 23:44:22.683909 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.683913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.683978 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.684077 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.685364 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.686012 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.687813 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.688032 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.688137 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.692189 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:33051: BlockingRecv error: recv got EOF from 127.21.112.66:33051 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:44:22.692834 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.682171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:33051)
0928 23:44:22.682817 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.682859 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.683652 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.684254 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.684262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.684709 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.685205 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.685218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.686148 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.686153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.687666 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.687681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.688206 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.688213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.688465 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.689151 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.689178 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.691967 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":102,"threads_started":1}
W20260928 23:44:22.693487 30976 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:33855 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.112.126:33855: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.699739 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.694009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:37019)
0928 23:44:22.694261 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.694274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.694339 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.694671 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.694673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.694815 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.694911 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.694915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.696033 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.696035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.696586 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.696594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.697157 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.697161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.697232 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.697761 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.697773 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.699631 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260928 23:44:22.699935 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.694135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:37019 (local address 127.21.112.126:33855)
0928 23:44:22.694291 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.694297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.694317 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:44:22.694419 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.694424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.694482 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.694643 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.695030 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.695915 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.696696 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.697054 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.697154 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.699813 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:37019: BlockingRecv error: recv got EOF from 127.21.112.66:37019 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.705585 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.700450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:58545)
0928 23:44:22.700645 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.700657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.700716 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.700928 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.700930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.701045 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.701112 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.701116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.702172 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.702176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.702711 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.702719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.703297 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.703300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.703357 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.703850 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.703861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.705443 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
W20260928 23:44:22.705762 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.700555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:58545 (local address 127.21.112.126:33855)
0928 23:44:22.700706 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.700713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.700729 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:44:22.700772 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.700775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.700816 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.700908 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.701213 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.702032 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.702843 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.703184 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.703287 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.705639 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:58545: BlockingRecv error: recv got EOF from 127.21.112.66:58545 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 23:44:22.705936 30976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:44:22.802928 30979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:22.803205 30979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:22.803249 30979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:22.807312 30979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:22.807375 30979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:22.807473 30979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:22.812963 30979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/krb5kdc/oryx_127.21.112.67.keytab
--principal=oryx/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:34977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=46113
--tserver_master_addrs=127.21.112.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:22.814092 30979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:22.815158 30979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:22.824595 30985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.824604 30987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:22.824615 30984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:22.824820 30979 server_base.cc:1034] running on GCE node
I20260928 23:44:22.825286 30979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:22.826020 30979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:22.827199 30979 hybrid_clock.cc:648] HybridClock initialized: now 1790639062827184 us; error 25 us; skew 500 ppm
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639062, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:22.829923 30979 init.cc:377] Logged in from keytab as oryx/127.21.112.67@KRBTEST.COM (short username oryx)
I20260928 23:44:22.830765 30979 webserver.cc:460] Webserver started at http://127.21.112.67:46113/ using document root <none> and password file <none>
I20260928 23:44:22.831311 30979 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:22.831408 30979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:22.835136 30979 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:22.837301 30994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.838526 30979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:44:22.838651 30979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "cbf334823fd84fdabba33774fc732347"
format_stamp: "Formatted at 2026-09-28 23:44:21 on dist-test-slave-c3s8"
server_key: "cc56f9b9dff67655d57bce94052df60b"
server_key_iv: "76f0804591b0a84f064957791a419cd6"
server_key_version: "encryptionkey@0"
I20260928 23:44:22.839288 30979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:22.862347 30979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:22.865267 30979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:22.865476 30979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:22.866098 30979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:22.866946 30979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:22.866992 30979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.867062 30979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:22.867094 30979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:22.876297 30979 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:34977
I20260928 23:44:22.876322 31107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:34977 every 8 connection(s)
I20260928 23:44:22.877331 30979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:22.881620 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 30979
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:44:22.888841 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.879248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:46611 (local address 127.21.112.126:33855)
0928 23:44:22.879395 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.879404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.880373 (+ 969us) server_negotiation.cc:407] Connection header received
0928 23:44:22.881309 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.881314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.881380 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.881483 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.882991 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.883521 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.884666 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.884895 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.885006 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.888702 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:46611: BlockingRecv error: recv got EOF from 127.21.112.67:46611 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:44:22.889691 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.879569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:46611)
0928 23:44:22.880231 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.880271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.881060 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.881735 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.881762 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.882185 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.882846 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.882861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.883651 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.883654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.884539 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.884548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.885046 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.885050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.885221 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.885845 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.885866 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.888602 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":120,"threads_started":1}
W20260928 23:44:22.890161 31108 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:33855 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.112.126:33855: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
I20260928 23:44:22.891690 31112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.882740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.21.112.126:33855)
0928 23:44:22.882999 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.883002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.883012 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:44:22.883042 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.883044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.883076 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.883172 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.883899 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.884420 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.885284 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.885615 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.885723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.887160 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:22.887166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:22.887179 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:22.887206 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:22.889072 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:22.889604 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:22.889609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:22.889615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:22.889730 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:22.890163 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:22.890167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:22.890169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:22.890629 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:22.890844 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:22.891149 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 23:44:22.891563 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":123,"threads_started":1}
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.896370 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.890635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:52729)
0928 23:44:22.890880 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.890894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.890975 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.891218 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.891221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.891395 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.891505 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.891510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.892480 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.892483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.893232 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.893242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.893761 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.893764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.893859 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.894359 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.894377 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.896279 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185}
W20260928 23:44:22.896543 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.890762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:52729 (local address 127.21.112.126:33855)
0928 23:44:22.890881 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.890886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.890937 (+ 51us) server_negotiation.cc:407] Connection header received
0928 23:44:22.891075 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.891078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.891116 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.891202 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.891612 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.892374 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.893348 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.893614 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.893722 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.896398 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:52729: BlockingRecv error: recv got EOF from 127.21.112.67:52729 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:22 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:22.901656 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:22.897212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:43173)
0928 23:44:22.897307 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:22.897322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:22.897392 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:22.897667 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:22.897670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:22.897793 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:22.897863 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.897868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.898545 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.898547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.899060 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:22.899067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.899467 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.899469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:22.899524 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.899893 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:22.899903 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:22.901497 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 23:44:22.901815 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:22.897320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43173 (local address 127.21.112.126:33855)
0928 23:44:22.897449 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:22.897458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:22.897469 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:44:22.897498 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:22.897500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:22.897569 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:22.897652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:22.897954 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.898448 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.899159 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:22.899375 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:22.899446 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:22.901665 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:43173: BlockingRecv error: recv got EOF from 127.21.112.67:43173 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 23:44:22.901866 31108 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:44:23 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:23 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:23.529971 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:23.523686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:52835)
0928 23:44:23.524089 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:23.524109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:23.524253 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:23.524531 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:23.524533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:23.524730 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:23.524829 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:23.524834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.525700 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.525705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:23.526635 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:23.526646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.527228 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.527232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:23.527320 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:23.527841 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:23.527853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:23.529807 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":149,"threads_started":1}
W20260928 23:44:23.530084 31114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:23.523918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:52835 (local address 127.21.112.126:33855)
0928 23:44:23.524191 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:23.524195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:23.524209 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:23.524356 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:23.524358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:23.524402 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:23.524486 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:23.524951 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.525518 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.526764 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.527095 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.527224 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:23.529952 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:52835: BlockingRecv error: recv got EOF from 127.21.112.65:52835 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
Sep 28 23:44:23 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:23 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:44:23.712740 31114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:23.706739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:58865 (local address 127.21.112.126:33855)
0928 23:44:23.706889 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:23.706894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:23.707061 (+ 167us) server_negotiation.cc:407] Connection header received
0928 23:44:23.707229 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:23.707234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:23.707283 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:23.707370 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:23.707826 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.708385 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.709264 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.709485 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.709675 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:23.712644 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:58865: BlockingRecv error: recv got EOF from 127.21.112.66:58865 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:44:23.712746 31115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:23.706672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:58865)
0928 23:44:23.706962 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:23.706984 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:23.707111 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:23.707390 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:23.707393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:23.707595 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:23.707693 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:23.707702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.708501 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.708506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:23.709101 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:23.709110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.709675 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.709678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:23.709769 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:23.710335 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:23.710365 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:23.712511 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
Sep 28 23:44:23 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:23 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:23.908196 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:23.902509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:41161)
0928 23:44:23.902790 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:23.902804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:23.902908 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:23.903178 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:23.903181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:23.903364 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:23.903461 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:23.903467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.904301 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.904304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:23.904898 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:23.904908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.905435 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.905437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:23.905505 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:23.906016 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:23.906028 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:23.908012 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
W20260928 23:44:23.908272 31114 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:23.902576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:41161 (local address 127.21.112.126:33855)
0928 23:44:23.902748 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:23.902760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:23.902915 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:44:23.903005 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:23.903008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:23.903056 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:23.903120 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:23.903581 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.904097 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.905030 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:23.905314 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:23.905401 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:23.908152 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:41161: BlockingRecv error: recv got EOF from 127.21.112.67:41161 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 23:44:24 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:24 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:24.536787 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:24.530819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:47199)
0928 23:44:24.531096 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:24.531114 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:24.531232 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:24.531586 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:24.531589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:24.531783 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:24.531883 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:24.531888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.532668 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.532670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:24.533270 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:24.533279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.533863 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.533866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:24.533936 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:24.534463 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:24.534475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:24.536594 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260928 23:44:24.537130 31118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:24.530981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:47199 (local address 127.21.112.126:33855)
0928 23:44:24.531303 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:24.531314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:24.531328 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:44:24.531365 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:24.531367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:24.531407 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:24.531500 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:24.532003 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.532526 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.533403 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.533711 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.533824 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:24.536800 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:47199: BlockingRecv error: recv got EOF from 127.21.112.65:47199 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":83,"threads_started":1}
Sep 28 23:44:24 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:24 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:44:24.720228 31118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:24.713766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:46557 (local address 127.21.112.126:33855)
0928 23:44:24.713922 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:24.713926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:24.714125 (+ 199us) server_negotiation.cc:407] Connection header received
0928 23:44:24.714243 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:24.714246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:24.714294 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:24.714372 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:24.714929 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.715467 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.716427 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.716663 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.716805 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:24.720121 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:46557: BlockingRecv error: recv got EOF from 127.21.112.66:46557 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:44:24.720244 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:24.713714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:46557)
0928 23:44:24.714026 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:24.714050 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:24.714159 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:24.714454 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:24.714457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:24.714668 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:24.714780 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:24.714786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.715608 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.715611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:24.716302 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:24.716313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.716815 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.716818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:24.716908 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:24.717659 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:24.717682 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:24.719966 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
Sep 28 23:44:24 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:24 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:24.920892 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:24.909133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:37493)
0928 23:44:24.909488 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:24.909507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:24.909652 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:24.909983 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:24.909986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:24.910302 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:24.910445 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:24.910453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.911433 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.911436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:24.912008 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:24.912017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.912445 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.912447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:24.912511 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:24.918712 (+ 6201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:24.918725 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:24.920746 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
W20260928 23:44:24.921126 31118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:24.909332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:37493 (local address 127.21.112.126:33855)
0928 23:44:24.909493 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:24.909498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:24.909606 (+ 108us) server_negotiation.cc:407] Connection header received
0928 23:44:24.909792 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:24.909795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:24.909853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:24.909944 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:24.910580 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.911306 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.912112 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:24.912345 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:24.912428 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:24.920970 (+ 8542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:37493: BlockingRecv error: recv got EOF from 127.21.112.67:37493 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 23:44:25 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:25 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:25.543360 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:25.537703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:57971)
0928 23:44:25.537993 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:25.538007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:25.538112 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:25.538419 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:25.538421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:25.538608 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:25.538707 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:25.538712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.539526 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.539529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:25.540085 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:25.540093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.540562 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.540564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:25.540630 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:25.541133 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:25.541144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:25.543165 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
W20260928 23:44:25.543556 31121 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:25.537720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:57971 (local address 127.21.112.126:33855)
0928 23:44:25.537995 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:25.537998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:25.538084 (+ 86us) server_negotiation.cc:407] Connection header received
0928 23:44:25.538248 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:25.538251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:25.538295 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:25.538376 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:25.538834 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.539416 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.540190 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.540455 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.540537 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:25.543409 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:57971: BlockingRecv error: recv got EOF from 127.21.112.65:57971 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
Sep 28 23:44:25 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:25 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:25.727211 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:25.721347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:42831)
0928 23:44:25.721668 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:25.721693 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:25.721808 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:25.722128 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:25.722131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:25.722385 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:25.722484 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:25.722490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.723361 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.723367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:25.723924 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:25.723932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.724415 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.724418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:25.724509 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:25.725090 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:25.725102 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:25.727024 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
W20260928 23:44:25.727265 31121 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:25.721419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:42831 (local address 127.21.112.126:33855)
0928 23:44:25.721616 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:25.721620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:25.721801 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:44:25.721920 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:25.721923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:25.721987 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:25.722083 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:25.722618 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.723242 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.724047 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.724289 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.724412 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:25.727158 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:42831: BlockingRecv error: recv got EOF from 127.21.112.66:42831 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 23:44:25 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:25 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:25.927505 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:25.921863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:44161)
0928 23:44:25.922142 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:25.922156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:25.922250 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:25.922509 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:25.922511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:25.922690 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:25.922786 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:25.922792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.923543 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.923546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:25.924099 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:25.924107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.924560 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.924562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:25.924623 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:25.925204 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:25.925221 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:25.927335 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
W20260928 23:44:25.927564 31121 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:25.922035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:44161 (local address 127.21.112.126:33855)
0928 23:44:25.922177 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:25.922181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:25.922221 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:44:25.922351 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:25.922353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:25.922404 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:25.922473 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:25.922910 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.923434 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.924209 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:25.924447 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:25.924529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:25.927456 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:44161: BlockingRecv error: recv got EOF from 127.21.112.67:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 23:44:26 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:26 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:26.550987 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:26.544348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:45043)
0928 23:44:26.544618 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:26.544639 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:26.544734 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:26.545149 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:26.545152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:26.545352 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:26.545455 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:26.545461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.546921 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.546926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:26.547758 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:26.547770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.548373 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.548376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:26.548454 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:26.548982 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:26.548994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:26.550866 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
W20260928 23:44:26.551172 31126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:26.544434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45043 (local address 127.21.112.126:33855)
0928 23:44:26.544689 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:26.544692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:26.544708 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:44:26.544860 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:26.544863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:26.544916 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:26.545009 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:26.545663 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.546708 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.547886 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.548166 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.548326 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:26.551018 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:45043: BlockingRecv error: recv got EOF from 127.21.112.65:45043 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
Sep 28 23:44:26 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:26 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:26.733695 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:26.728086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:36491)
0928 23:44:26.728399 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:26.728413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:26.728520 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:26.728734 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:26.728736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:26.728923 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:26.729029 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:26.729034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.729921 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.729928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:26.730512 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:26.730526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.730976 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.730979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:26.731043 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:26.731588 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:26.731600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:26.733502 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":121,"threads_started":1}
W20260928 23:44:26.733798 31126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:26.728176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:36491 (local address 127.21.112.126:33855)
0928 23:44:26.728333 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:26.728338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:26.728501 (+ 163us) server_negotiation.cc:407] Connection header received
0928 23:44:26.728586 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:26.728588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:26.728631 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:26.728699 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:26.729159 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.729799 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.730653 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.730867 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.730952 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:26.733686 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:36491: BlockingRecv error: recv got EOF from 127.21.112.66:36491 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 23:44:26 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:26 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:26.934481 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:26.928450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:50085)
0928 23:44:26.928777 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:26.928792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:26.928900 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:26.929146 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:26.929148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:26.929335 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:26.929435 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:26.929440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.930197 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.930201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:26.930835 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:26.930844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.931383 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.931386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:26.931456 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:26.932191 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:26.932206 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:26.934368 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":116,"threads_started":1}
W20260928 23:44:26.934623 31126 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:26.928650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:50085 (local address 127.21.112.126:33855)
0928 23:44:26.928788 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:26.928792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:26.928897 (+ 105us) server_negotiation.cc:407] Connection header received
0928 23:44:26.928975 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:26.928977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:26.929024 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:26.929101 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:26.929562 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.930082 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.930970 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:26.931252 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:26.931347 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:26.934502 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:50085: BlockingRecv error: recv got EOF from 127.21.112.67:50085 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 23:44:27 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:27 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:27.557844 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:27.551896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:55991)
0928 23:44:27.552222 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:27.552247 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:27.552356 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:27.552616 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:27.552618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:27.552814 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:27.552908 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:27.552914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.553888 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.553893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:27.554505 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:27.554515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.554997 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.555001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:27.555089 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:27.555610 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:27.555621 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:27.557663 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
W20260928 23:44:27.558010 31130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:27.551964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:55991 (local address 127.21.112.126:33855)
0928 23:44:27.552235 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:27.552240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:27.552323 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:44:27.552437 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:27.552439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:27.552483 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:27.552563 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:27.553027 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.553734 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.554632 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.554879 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.554992 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:27.557835 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:55991: BlockingRecv error: recv got EOF from 127.21.112.65:55991 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":64,"threads_started":1}
Sep 28 23:44:27 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:27 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:27.740880 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:27.734621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:45273)
0928 23:44:27.734957 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:27.734975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:27.735092 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:27.735417 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:27.735421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:27.735640 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:27.735727 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:27.735733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.736683 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.736686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:27.737421 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:27.737431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.737999 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.738020 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:27.738105 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:27.738629 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:27.738642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:27.740754 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
W20260928 23:44:27.741112 31130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:27.734846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:45273 (local address 127.21.112.126:33855)
0928 23:44:27.735012 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:27.735019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:27.735036 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:27.735209 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:27.735212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:27.735266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:27.735399 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:27.735905 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.736558 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.737561 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.737873 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.738005 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:27.740960 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:45273: BlockingRecv error: recv got EOF from 127.21.112.66:45273 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 23:44:27 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:27 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:27.941640 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:27.935519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:33073)
0928 23:44:27.935927 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:27.935946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:27.936108 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:27.936432 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:27.936435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:27.936708 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:27.936873 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:27.936885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.937775 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.937778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:27.938380 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:27.938389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.938883 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.938887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:27.938972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:27.939513 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:27.939525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:27.941443 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
W20260928 23:44:27.941787 31130 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:27.935747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:33073 (local address 127.21.112.126:33855)
0928 23:44:27.935928 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:27.935935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:27.936052 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:44:27.936235 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:27.936238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:27.936303 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:27.936413 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:27.936985 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.937612 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.938506 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:27.938759 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:27.938879 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:27.941618 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:33073: BlockingRecv error: recv got EOF from 127.21.112.67:33073 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 23:44:28 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:28 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:28.564407 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:28.558844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:55147)
0928 23:44:28.559111 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:28.559125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:28.559214 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:28.559447 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:28.559449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:28.559636 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:28.559730 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:28.559735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.560574 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.560577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:28.561168 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:28.561176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.561728 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.561732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:28.561802 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:28.562473 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:28.562485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:28.564287 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
W20260928 23:44:28.564659 31134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:28.558934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:55147 (local address 127.21.112.126:33855)
0928 23:44:28.559168 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:28.559172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:28.559191 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:44:28.559283 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:28.559285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:28.559327 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:28.559406 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:28.559869 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.560463 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.561334 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.561592 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.561683 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:28.564495 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:55147: BlockingRecv error: recv got EOF from 127.21.112.65:55147 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
Sep 28 23:44:28 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:28 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:28.747470 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:28.741725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:37755)
0928 23:44:28.742035 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:28.742049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:28.742153 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:28.742401 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:28.742403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:28.742640 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:28.742737 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:28.742743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.743545 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.743547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:28.744195 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:28.744205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.744743 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.744747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:28.744875 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:28.745463 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:28.745484 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:28.747352 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":135,"threads_started":1}
W20260928 23:44:28.747663 31134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:28.741819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:37755 (local address 127.21.112.126:33855)
0928 23:44:28.741956 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:28.741960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:28.742119 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:44:28.742234 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:28.742236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:28.742280 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:28.742362 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:28.742852 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.743438 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.744314 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.744610 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.744733 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:28.747498 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:37755: BlockingRecv error: recv got EOF from 127.21.112.66:37755 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 23:44:28 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:28 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:28.948354 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:28.942562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:51337)
0928 23:44:28.942873 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:28.942888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:28.942996 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:28.943373 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:28.943375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:28.943558 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:28.943648 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:28.943654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.944543 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.944546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:28.945097 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:28.945105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.945525 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.945528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:28.945669 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:28.946244 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:28.946256 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:28.948159 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":130,"threads_started":1}
W20260928 23:44:28.948421 31134 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:28.942675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:51337 (local address 127.21.112.126:33855)
0928 23:44:28.942807 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:28.942811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:28.942963 (+ 152us) server_negotiation.cc:407] Connection header received
0928 23:44:28.943169 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:28.943174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:28.943237 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:28.943362 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:28.943783 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.944429 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.945207 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:28.945425 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:28.945511 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:28.948305 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:51337: BlockingRecv error: recv got EOF from 127.21.112.67:51337 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 23:44:29 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:29 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:29.571542 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:29.565315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:44161)
0928 23:44:29.565683 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:29.565697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:29.565794 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:29.566037 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:29.566040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:29.566225 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:29.566365 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:29.566372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.567137 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.567139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:29.567900 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:29.567913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.568524 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.568528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:29.568635 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:29.569194 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:29.569210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:29.571364 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
W20260928 23:44:29.571602 31138 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:29.565492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:44161 (local address 127.21.112.126:33855)
0928 23:44:29.565763 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:29.565767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:29.565783 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:44:29.565870 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:29.565872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:29.565915 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:29.566004 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:29.566493 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.567038 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.568049 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.568393 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.568507 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:29.571481 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:44161: BlockingRecv error: recv got EOF from 127.21.112.65:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
Sep 28 23:44:29 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:29 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:29.754359 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:29.748421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:41183)
0928 23:44:29.748763 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:29.748777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:29.748902 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:29.749194 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:29.749198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:29.749407 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:29.749504 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:29.749510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.750379 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.750382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:29.750971 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:29.750980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.751510 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.751515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:29.751612 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:29.752162 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:29.752183 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:29.754233 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
W20260928 23:44:29.754500 31138 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:29.748486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:41183 (local address 127.21.112.126:33855)
0928 23:44:29.748653 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:29.748656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:29.748852 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:44:29.748990 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:29.748992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:29.749039 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:29.749112 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:29.749705 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.750254 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.751097 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.751349 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.751448 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:29.754379 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:41183: BlockingRecv error: recv got EOF from 127.21.112.66:41183 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 23:44:29 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:29 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:29.959759 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:29.949314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:57951)
0928 23:44:29.949657 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:29.949670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:29.949771 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:29.950024 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:29.950026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:29.950210 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:29.950329 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:29.950334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.951313 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.951317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:29.952021 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:29.952036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.952655 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.952657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:29.952722 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:29.957608 (+ 4886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:29.957622 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:29.959639 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
W20260928 23:44:29.960047 31138 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:29.949481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:57951 (local address 127.21.112.126:33855)
0928 23:44:29.949644 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:29.949648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:29.949765 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:44:29.949858 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:29.949860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:29.949910 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:29.949990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:29.950465 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.951184 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.952173 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:29.952544 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:29.952640 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:29.959902 (+ 7262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:57951: BlockingRecv error: recv got EOF from 127.21.112.67:57951 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 23:44:30 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:30 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:30.579416 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:30.572545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:60093)
0928 23:44:30.573022 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:30.573047 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:30.573231 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:30.573600 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:30.573603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:30.573804 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:30.573908 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:30.573914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.575243 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.575246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:30.576065 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:30.576076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.576540 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.576543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:30.576614 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:30.577135 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:30.577147 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:30.579286 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":86,"threads_started":1}
W20260928 23:44:30.579703 31142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:30.572725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:60093 (local address 127.21.112.126:33855)
0928 23:44:30.573077 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:30.573082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:30.573099 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:30.573391 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:30.573394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:30.573457 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:30.573573 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:30.574028 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.575018 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.576171 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.576419 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.576513 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:30.579500 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:60093: BlockingRecv error: recv got EOF from 127.21.112.65:60093 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
Sep 28 23:44:30 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:30 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:30.762466 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:30.755409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:36581)
0928 23:44:30.755769 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:30.755783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:30.755897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:30.756399 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:30.756402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:30.756586 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:30.756683 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:30.756689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.757902 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.757905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:30.758475 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:30.758484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.759056 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.759060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:30.759132 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:30.759723 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:30.759735 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:30.762232 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":137,"threads_started":1}
W20260928 23:44:30.762880 31142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:30.755417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:36581 (local address 127.21.112.126:33855)
0928 23:44:30.755601 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:30.755607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:30.755885 (+ 278us) server_negotiation.cc:407] Connection header received
0928 23:44:30.756096 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:30.756104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:30.756176 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:30.756258 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:30.756838 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.757672 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.758613 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.758890 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.758996 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:30.762779 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:36581: BlockingRecv error: recv got EOF from 127.21.112.66:36581 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 23:44:30 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:30 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:30.966504 31144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:30.960536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:42477)
0928 23:44:30.960862 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:30.960884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:30.961019 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:30.961290 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:30.961293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:30.961475 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:30.961599 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:30.961605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.962360 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.962363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:30.962957 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:30.962966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.963539 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.963542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:30.963625 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:30.964151 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:30.964163 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:30.966329 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":83,"threads_started":1}
W20260928 23:44:30.966573 31142 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:30.960624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:42477 (local address 127.21.112.126:33855)
0928 23:44:30.960816 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:30.960821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:30.961050 (+ 229us) server_negotiation.cc:407] Connection header received
0928 23:44:30.961111 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:30.961113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:30.961169 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:30.961250 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:30.961723 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.962256 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.963087 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:30.963369 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:30.963468 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:30.966460 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:42477: BlockingRecv error: recv got EOF from 127.21.112.67:42477 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 23:44:31 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:31 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:31.586830 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:31.580454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:44651)
0928 23:44:31.580892 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:31.580932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:31.581067 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:31.581407 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:31.581410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:31.581717 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:31.581869 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:31.581898 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.582718 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.582721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:31.583394 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:31.583404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.583890 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.583893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:31.583961 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:31.584451 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:31.584463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:31.586623 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":91,"threads_started":1}
W20260928 23:44:31.586894 31146 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:31.580608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:44651 (local address 127.21.112.126:33855)
0928 23:44:31.580857 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:31.580882 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:31.581066 (+ 184us) server_negotiation.cc:407] Connection header received
0928 23:44:31.581199 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:31.581201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:31.581253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:31.581342 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:31.582030 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.582608 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.583532 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.583781 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.583870 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:31.586761 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:44651: BlockingRecv error: recv got EOF from 127.21.112.65:44651 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
Sep 28 23:44:31 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:31 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:31.769328 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:31.763477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:35681)
0928 23:44:31.763756 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:31.763777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:31.763878 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:31.764163 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:31.764166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:31.764381 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:31.764478 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:31.764484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.765246 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.765249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:31.765851 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:31.765859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.766435 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.766438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:31.766503 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:31.767062 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:31.767074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:31.769161 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
W20260928 23:44:31.769474 31146 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:31.763533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:35681 (local address 127.21.112.126:33855)
0928 23:44:31.763688 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:31.763692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:31.763847 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:44:31.763970 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:31.763972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:31.764020 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:31.764081 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:31.764615 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.765145 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.766005 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.766302 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.766405 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:31.769356 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:35681: BlockingRecv error: recv got EOF from 127.21.112.66:35681 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 23:44:31 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:31 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:31.973587 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:31.967613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:52357)
0928 23:44:31.967913 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:31.967932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:31.968053 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:31.968375 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:31.968378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:31.968598 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:31.968738 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:31.968747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.969516 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.969521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:31.970148 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:31.970156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.970628 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.970632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:31.970718 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:31.971286 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:31.971297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:31.973396 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
W20260928 23:44:31.973810 31146 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:31.967770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:52357 (local address 127.21.112.126:33855)
0928 23:44:31.967963 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:31.967967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:31.967988 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:44:31.968177 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:31.968180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:31.968237 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:31.968346 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:31.968872 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.969404 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.970277 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:31.970513 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:31.970623 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:31.973662 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:52357: BlockingRecv error: recv got EOF from 127.21.112.67:52357 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 23:44:32 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:32 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:32.593775 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:32.587783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:53135)
0928 23:44:32.588071 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:32.588084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:32.588174 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:32.588524 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:32.588526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:32.588725 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:32.588819 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:32.588824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.589652 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.589659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:32.590393 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:32.590406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.591023 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.591027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:32.591102 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:32.591601 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:32.591613 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:32.593655 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
W20260928 23:44:32.593967 31150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:32.587979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:53135 (local address 127.21.112.126:33855)
0928 23:44:32.588233 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:32.588237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:32.588255 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:44:32.588303 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:32.588306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:32.588348 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:32.588485 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:32.588963 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.589507 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.590550 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.590885 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.591002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:32.593805 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:53135: BlockingRecv error: recv got EOF from 127.21.112.65:53135 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
Sep 28 23:44:32 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:32 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:32.776937 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:32.770379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:59461)
0928 23:44:32.770766 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:32.770784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:32.770895 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:32.771175 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:32.771180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:32.771486 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:32.771613 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:32.771619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.772698 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.772702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:32.773377 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:32.773387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.773991 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.773998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:32.774063 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:32.774674 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:32.774686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:32.776797 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":114,"threads_started":1}
W20260928 23:44:32.777127 31150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:32.770383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:59461 (local address 127.21.112.126:33855)
0928 23:44:32.770538 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:32.770543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:32.770876 (+ 333us) server_negotiation.cc:407] Connection header received
0928 23:44:32.770973 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:32.770977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:32.771036 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:32.771148 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:32.771813 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.772564 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.773520 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.773875 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.773969 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:32.776971 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:59461: BlockingRecv error: recv got EOF from 127.21.112.66:59461 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 23:44:32 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:32 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:32.980446 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:32.974465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:45681)
0928 23:44:32.974780 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:32.974795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:32.974889 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:32.975247 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:32.975250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:32.975446 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:32.975541 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:32.975547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.976375 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.976378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:32.977053 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:32.977063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.977622 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.977629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:32.977709 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:32.978316 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:32.978328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:32.980326 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20260928 23:44:32.980669 31150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:32.974652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:45681 (local address 127.21.112.126:33855)
0928 23:44:32.974797 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:32.974801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:32.974856 (+ 55us) server_negotiation.cc:407] Connection header received
0928 23:44:32.975017 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:32.975020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:32.975068 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:32.975143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:32.975673 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.976244 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.977188 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:32.977422 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:32.977509 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:32.980534 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:45681: BlockingRecv error: recv got EOF from 127.21.112.67:45681 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 23:44:33 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:33 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:33.601349 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:33.594715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:38345)
0928 23:44:33.595062 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:33.595077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:33.595233 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:33.595569 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:33.595571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:33.595763 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:33.595890 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:33.595896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.596942 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.596946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:33.597828 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:33.597845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.598482 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.598486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:33.598587 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:33.599211 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:33.599226 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:33.601226 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
W20260928 23:44:33.601758 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:33.594911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:38345 (local address 127.21.112.126:33855)
0928 23:44:33.595127 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:33.595131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:33.595146 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:33.595383 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:33.595385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:33.595436 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:33.595524 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:33.596101 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.596773 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.597975 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.598291 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.598436 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:33.601413 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:38345: BlockingRecv error: recv got EOF from 127.21.112.65:38345 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
Sep 28 23:44:33 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:33 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:33.783561 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:33.777943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:36301)
0928 23:44:33.778280 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:33.778291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:33.778423 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:33.778714 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:33.778717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:33.778936 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:33.779029 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:33.779035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.779800 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.779805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:33.780377 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:33.780386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.780862 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.780865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:33.780960 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:33.781525 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:33.781568 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:33.783380 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":93,"threads_started":1}
W20260928 23:44:33.783643 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:33.778004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:36301 (local address 127.21.112.126:33855)
0928 23:44:33.778171 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:33.778176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:33.778370 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:44:33.778534 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:33.778539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:33.778595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:33.778687 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:33.779152 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.779684 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.780505 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.780743 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.780857 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:33.783525 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:36301: BlockingRecv error: recv got EOF from 127.21.112.66:36301 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 23:44:33 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:33 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:33.987052 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:33.981351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:54077)
0928 23:44:33.981709 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:33.981750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:33.981864 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:33.982151 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:33.982153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:33.982373 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:33.982464 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:33.982469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.983235 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.983238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:33.983807 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:33.983815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.984411 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.984415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:33.984504 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:33.985055 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:33.985067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:33.986883 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":97,"threads_started":1}
W20260928 23:44:33.987114 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:33.981687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:54077 (local address 127.21.112.126:33855)
0928 23:44:33.981836 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:33.981841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:33.981854 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:44:33.981977 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:33.981984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:33.982039 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:33.982128 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:33.982584 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.983127 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.983930 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:33.984171 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:33.984413 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:33.987001 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:54077: BlockingRecv error: recv got EOF from 127.21.112.67:54077 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 23:44:34 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:34 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:34.608348 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:34.602510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:34433)
0928 23:44:34.602853 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:34.602872 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:34.603013 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:34.603341 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:34.603344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:34.603536 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:34.603644 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:34.603649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.604497 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.604503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:34.605070 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:34.605078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.605589 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.605592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:34.605682 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:34.606278 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:34.606290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:34.608175 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
W20260928 23:44:34.608458 31158 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:34.602601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:34433 (local address 127.21.112.126:33855)
0928 23:44:34.602874 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:34.602879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:34.602946 (+ 67us) server_negotiation.cc:407] Connection header received
0928 23:44:34.603149 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:34.603151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:34.603199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:34.603277 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:34.603780 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.604383 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.605194 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.605437 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.605576 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:34.608321 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:34433: BlockingRecv error: recv got EOF from 127.21.112.65:34433 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
Sep 28 23:44:34 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:34 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:34.789932 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:34.784395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:58003)
0928 23:44:34.784673 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:34.784687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:34.784778 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:34.785022 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:34.785024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:34.785201 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:34.785294 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:34.785299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.786099 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.786102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:34.786699 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:34.786708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.787131 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.787134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:34.787194 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:34.787719 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:34.787731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:34.789751 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":70,"threads_started":1}
W20260928 23:44:34.790009 31158 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:34.784548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:58003 (local address 127.21.112.126:33855)
0928 23:44:34.784710 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:34.784714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:34.784731 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:34.784865 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:34.784867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:34.784914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:34.784982 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:34.785427 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.786004 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.786812 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.787035 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.787110 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:34.789888 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:58003: BlockingRecv error: recv got EOF from 127.21.112.66:58003 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 23:44:34 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:34 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:35.004463 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:34.988104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:44217)
0928 23:44:34.988394 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:34.988407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:34.988545 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:34.988823 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:34.988826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:34.989040 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:34.989130 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:34.989136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.990019 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.990024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:34.990595 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:34.990604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.991064 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.991068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:34.991146 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:35.002227 (+ 11081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:35.002242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:35.004326 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260928 23:44:35.004654 31158 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:34.988164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:44217 (local address 127.21.112.126:33855)
0928 23:44:34.988332 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:34.988337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:34.988486 (+ 149us) server_negotiation.cc:407] Connection header received
0928 23:44:34.988653 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:34.988656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:34.988711 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:34.988798 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:34.989254 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.989906 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.990719 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:34.990951 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:34.991062 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:35.004510 (+ 13448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:44217: BlockingRecv error: recv got EOF from 127.21.112.67:44217 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 23:44:35 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:35 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:35.615913 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:35.609417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:57485)
0928 23:44:35.609849 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:35.609870 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:35.610048 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:35.610645 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:35.610648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:35.610840 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:35.610944 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:35.610950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.611803 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.611806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:35.612435 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:35.612445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.612935 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.612938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:35.613009 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:35.613686 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:35.613700 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:35.615782 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":114,"threads_started":1}
W20260928 23:44:35.616267 31161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:35.609419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:57485 (local address 127.21.112.126:33855)
0928 23:44:35.610269 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:35.610275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:35.610291 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:44:35.610339 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:35.610345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:35.610393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:35.610488 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:35.611075 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.611645 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.612568 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.612797 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.612883 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:35.615953 (+ 3070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:57485: BlockingRecv error: recv got EOF from 127.21.112.65:57485 (error 108)
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":114,"threads_started":1}
Sep 28 23:44:35 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:35 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:35.796766 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:35.790831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:50027)
0928 23:44:35.791170 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:35.791187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:35.791416 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:35.791730 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:35.791733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:35.791983 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:35.792074 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:35.792081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.792870 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.792875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:35.793451 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:35.793459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.794037 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.794041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:35.794127 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:35.794717 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:35.794729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:35.796578 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
W20260928 23:44:35.796820 31161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:35.791009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:50027 (local address 127.21.112.126:33855)
0928 23:44:35.791172 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:35.791178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:35.791334 (+ 156us) server_negotiation.cc:407] Connection header received
0928 23:44:35.791554 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:35.791559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:35.791615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:35.791707 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:35.792209 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.792757 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.793672 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:35.793909 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:35.794033 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:35.796711 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:50027: BlockingRecv error: recv got EOF from 127.21.112.66:50027 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 23:44:36 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:36 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:36.012802 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:36.005502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:42557)
0928 23:44:36.006112 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:36.006126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:36.006229 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:36.006594 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:36.006597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:36.006860 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:36.007018 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:36.007030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.007993 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.007996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:36.008650 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:36.008659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.009118 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.009121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:36.009188 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:36.009856 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:36.009880 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:36.012586 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":119,"threads_started":1}
W20260928 23:44:36.012992 31161 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:36.005682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:42557 (local address 127.21.112.126:33855)
0928 23:44:36.005869 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:36.005875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:36.006208 (+ 333us) server_negotiation.cc:407] Connection header received
0928 23:44:36.006371 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:36.006374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:36.006433 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:36.006535 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:36.007190 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.007879 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.008769 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.009011 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.009095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:36.012823 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:42557: BlockingRecv error: recv got EOF from 127.21.112.67:42557 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 23:44:36 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:36 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:36.623005 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:36.616787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:45069)
0928 23:44:36.617154 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:36.617169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:36.617269 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:36.617698 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:36.617701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:36.617903 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:36.618014 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:36.618021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.618839 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.618842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:36.619517 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:36.619526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.620005 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.620008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:36.620075 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:36.620723 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:36.620736 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:36.622855 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":95,"threads_started":1}
W20260928 23:44:36.623173 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:36.616930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45069 (local address 127.21.112.126:33855)
0928 23:44:36.617237 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:36.617240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:36.617253 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:44:36.617420 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:36.617422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:36.617480 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:36.617618 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:36.618139 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.618725 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.619646 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.619885 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.619972 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:36.623014 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:45069: BlockingRecv error: recv got EOF from 127.21.112.65:45069 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
Sep 28 23:44:36 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:36 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:36.803500 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:36.797703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:38621)
0928 23:44:36.798002 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:36.798019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:36.798128 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:36.798463 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:36.798466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:36.798656 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:36.798759 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:36.798765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.799650 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.799653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:36.800253 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:36.800262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.800769 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.800772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:36.800833 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:36.801334 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:36.801345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:36.803335 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
W20260928 23:44:36.803673 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:36.797896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:38621 (local address 127.21.112.126:33855)
0928 23:44:36.798097 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:36.798101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:36.798116 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:36.798255 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:36.798257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:36.798304 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:36.798405 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:36.798920 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.799530 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.800421 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:36.800656 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:36.800751 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:36.803512 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:38621: BlockingRecv error: recv got EOF from 127.21.112.66:38621 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 28 23:44:37 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:37 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.67@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:37.023018 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:37.013838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.67:52991)
0928 23:44:37.014357 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:37.014373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:37.014493 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:37.014773 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:37.014817 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:37.015140 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:37.015340 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:37.015373 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.016960 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.016965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:37.018068 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:37.018086 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.019008 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.019015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:37.019152 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:37.019843 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:37.019866 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:37.022804 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":346,"threads_started":1}
W20260928 23:44:37.023069 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:37.013882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:52991 (local address 127.21.112.126:33855)
0928 23:44:37.014092 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:37.014100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:37.014466 (+ 366us) server_negotiation.cc:407] Connection header received
0928 23:44:37.014579 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:37.014582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:37.014661 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:37.014763 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:37.015619 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.016628 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.018397 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.018833 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.018967 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:37.022949 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.67:52991: BlockingRecv error: recv got EOF from 127.21.112.67:52991 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 23:44:37 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:37 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.65@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:37.630204 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:37.624026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.65:35823)
0928 23:44:37.624352 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:37.624366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:37.624464 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:37.624718 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:37.624720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:37.624932 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:37.625031 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:37.625037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.625785 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.625787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:37.626427 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:37.626438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.627080 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.627083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:37.627193 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:37.627812 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:37.627825 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:37.630076 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
W20260928 23:44:37.630385 31170 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:37.624159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:35823 (local address 127.21.112.126:33855)
0928 23:44:37.624405 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:37.624430 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:37.624445 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:44:37.624548 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:37.624550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:37.624592 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:37.624670 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:37.625140 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.625692 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.626546 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.626960 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.627066 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:37.630212 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.65:35823: BlockingRecv error: recv got EOF from 127.21.112.65:35823 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
Sep 28 23:44:37 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:44:37 dist-test-slave-c3s8 krb5kdc[30110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.112.66@KRBTEST.COM for oryx/127.21.112.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:44:37.810390 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:37.804557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33855 (local address 127.21.112.66:59413)
0928 23:44:37.804846 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:37.804860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:37.804953 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:37.805204 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:37.805206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:37.805431 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:37.805528 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:37.805567 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.806303 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.806306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:37.807084 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:37.807094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.807703 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.807707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:37.807797 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:37.808361 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:37.808375 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:37.810216 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.112.126:33855: Server oryx/127.21.112.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
W20260928 23:44:37.810452 31170 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:37.804769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:59413 (local address 127.21.112.126:33855)
0928 23:44:37.804914 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:37.804918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:37.804934 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:44:37.805045 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:37.805047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:37.805090 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:37.805170 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:37.805660 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.806175 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.807247 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:37.807582 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:37.807681 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:37.810335 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.112.66:59413: BlockingRecv error: recv got EOF from 127.21.112.66:59413 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:44:37.890196 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30695
I20260928 23:44:37.896018 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30845
I20260928 23:44:37.901230 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30979
I20260928 23:44:37.906430 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 30625
2026-09-28T23:44:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18128 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T23:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:37Z Disabled control of system clock
I20260928 23:44:37.938407 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34599
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:34599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:44:38.057417 31175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:38.057746 31175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:38.057782 31175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:38.061954 31175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:38.062023 31175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:38.062038 31175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:38.062055 31175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:38.062071 31175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:38.067091 31175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:34599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:38.068203 31175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:38.069622 31175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:38.079288 31180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.079391 31175 server_base.cc:1034] running on GCE node
W20260928 23:44:38.079349 31183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:38.079282 31181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.080101 31175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:38.081266 31175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:38.082516 31175 hybrid_clock.cc:648] HybridClock initialized: now 1790639078082485 us; error 48 us; skew 500 ppm
I20260928 23:44:38.085031 31175 webserver.cc:460] Webserver started at http://127.21.112.126:33599/ using document root <none> and password file <none>
I20260928 23:44:38.085865 31175 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:38.085959 31175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:38.086174 31175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:38.088339 31175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "ddf560c1fca64e48b6ff7875e685732c"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "f440a35b929a34c1d489b5cea545aadf"
server_key_iv: "4204f6877222c1de3c01ddd7c5d09771"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.088951 31175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "ddf560c1fca64e48b6ff7875e685732c"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "f440a35b929a34c1d489b5cea545aadf"
server_key_iv: "4204f6877222c1de3c01ddd7c5d09771"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.093041 31175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:44:38.096334 31189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.097894 31175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:38.098044 31175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "ddf560c1fca64e48b6ff7875e685732c"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "f440a35b929a34c1d489b5cea545aadf"
server_key_iv: "4204f6877222c1de3c01ddd7c5d09771"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.098173 31175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:38.110103 31175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:38.110934 31175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:38.111163 31175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:38.119197 31241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:34599 every 8 connection(s)
I20260928 23:44:38.119403 31175 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:34599
I20260928 23:44:38.120445 31175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:38.124060 31242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:38.124845 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31175
I20260928 23:44:38.125010 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:38.125308 21953 external_mini_cluster.cc:1428] Setting key de6a8971b8b01eebfea39fe48f6f80f5
I20260928 23:44:38.130952 31242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c: Bootstrap starting.
I20260928 23:44:38.133700 31242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:38.134598 31242 log.cc:824] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:38.135479 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.126809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.21.112.126:34599)
0928 23:44:38.127473 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:38.127488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:38.127524 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:44:38.128168 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:38.128187 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:38.128632 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:38.128994 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:38.129361 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.130409 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.131616 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.132127 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.132264 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.133197 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:38.133202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:38.133230 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:38.133795 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:38.133895 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:38.133931 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:38.134015 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:38.134185 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 23:44:38.134546 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
I20260928 23:44:38.136718 31242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c: No bootstrap required, opened a new log
I20260928 23:44:38.140323 31242 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf560c1fca64e48b6ff7875e685732c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34599 } }
I20260928 23:44:38.140549 31242 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:38.140617 31242 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf560c1fca64e48b6ff7875e685732c, State: Initialized, Role: FOLLOWER
I20260928 23:44:38.141054 31242 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf560c1fca64e48b6ff7875e685732c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34599 } }
I20260928 23:44:38.141202 31242 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:38.141273 31242 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:38.141368 31242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:38.142417 31242 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf560c1fca64e48b6ff7875e685732c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34599 } }
I20260928 23:44:38.142793 31242 leader_election.cc:302] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf560c1fca64e48b6ff7875e685732c; no voters:
I20260928 23:44:38.143134 31242 leader_election.cc:288] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:38.143265 31247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:38.143501 31247 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [term 1 LEADER]: Becoming Leader. State: Replica: ddf560c1fca64e48b6ff7875e685732c, State: Running, Role: LEADER
I20260928 23:44:38.143832 31247 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf560c1fca64e48b6ff7875e685732c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34599 } }
I20260928 23:44:38.144435 31242 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:38.145663 31248 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf560c1fca64e48b6ff7875e685732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf560c1fca64e48b6ff7875e685732c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34599 } } }
I20260928 23:44:38.145799 31248 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:38.146032 31249 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf560c1fca64e48b6ff7875e685732c. Latest consensus state: current_term: 1 leader_uuid: "ddf560c1fca64e48b6ff7875e685732c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf560c1fca64e48b6ff7875e685732c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34599 } } }
I20260928 23:44:38.146138 31249 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:38.146246 31256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:38.149145 31256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:38.155263 31256 catalog_manager.cc:1347] Generated new cluster ID: 88f3970d561147a8bc8a43020ba02ac3
I20260928 23:44:38.155458 31256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:38.174038 31256 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:38.175475 31256 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:38.182673 31256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf560c1fca64e48b6ff7875e685732c: Generated new TSK 0
I20260928 23:44:38.183502 31256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:44:38.190351 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34599
--builtin_ntp_servers=127.21.112.84:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:44:38.307785 31266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:38.308060 31266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:38.308099 31266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:38.312274 31266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:38.312346 31266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:38.312403 31266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:38.317413 31266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:38.318662 31266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:38.320099 31266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:38.330643 31274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:38.330718 31272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.330763 31266 server_base.cc:1034] running on GCE node
W20260928 23:44:38.330643 31271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.331480 31266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:38.332233 31266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:38.333451 31266 hybrid_clock.cc:648] HybridClock initialized: now 1790639078333420 us; error 41 us; skew 500 ppm
I20260928 23:44:38.335566 31266 webserver.cc:460] Webserver started at http://127.21.112.65:40317/ using document root <none> and password file <none>
I20260928 23:44:38.336220 31266 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:38.336316 31266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:38.336560 31266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:38.338829 31266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "24b6166ffc4941e69b868dce215a4eea"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "06db066caee973c3829533cfacaf5d08"
server_key_iv: "de2ae251e03a758b5cad97e79fc5d2fd"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.339443 31266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "24b6166ffc4941e69b868dce215a4eea"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "06db066caee973c3829533cfacaf5d08"
server_key_iv: "de2ae251e03a758b5cad97e79fc5d2fd"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.343475 31266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 23:44:38.346503 31280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.347875 31266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:38.348021 31266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "24b6166ffc4941e69b868dce215a4eea"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "06db066caee973c3829533cfacaf5d08"
server_key_iv: "de2ae251e03a758b5cad97e79fc5d2fd"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.348150 31266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:38.363679 31266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:38.364452 31266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:38.364662 31266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:38.365278 31266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:38.366317 31266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:38.366410 31266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.366510 31266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:38.366566 31266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.376235 31266 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:38031
I20260928 23:44:38.376255 31393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:38031 every 8 connection(s)
I20260928 23:44:38.377241 31266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:38.386935 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.379124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:57143 (local address 127.21.112.126:34599)
0928 23:44:38.379267 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:38.379271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:38.380500 (+ 1229us) server_negotiation.cc:407] Connection header received
0928 23:44:38.382036 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:38.382040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:38.382102 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:38.382214 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:38.383133 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.383695 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.384877 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.385089 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.385212 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.386399 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:38.386407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:38.386409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:38.386421 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:38.386440 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:38.386448 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:38.386542 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:38.386703 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 23:44:38.386772 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:44:38.387388 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31266
I20260928 23:44:38.387545 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:38.387840 21953 external_mini_cluster.cc:1428] Setting key 2cf12c4684c359e9a8bf19e586857722
I20260928 23:44:38.388077 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.379573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34599 (local address 127.21.112.65:57143)
0928 23:44:38.380322 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:38.380364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:38.381817 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:38.382391 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:38.382398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:38.382488 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:38.382985 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:38.382997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.383822 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.383826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:38.384716 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:38.384725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.385244 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.385248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:38.385433 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.386114 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:38.386116 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:38.386156 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:38.386219 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:38.386544 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:38.386556 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:38.386564 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:44:38.386703 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 23:44:38.387007 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":115,"threads_started":1}
I20260928 23:44:38.389441 31394 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34599
I20260928 23:44:38.389704 31394 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:38.390027 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34599
--builtin_ntp_servers=127.21.112.84:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:44:38.390368 31394 heartbeater.cc:511] Master 127.21.112.126:34599 requested a full tablet report, sending...
I20260928 23:44:38.392109 31206 ts_manager.cc:194] Registered new tserver with Master: 24b6166ffc4941e69b868dce215a4eea (127.21.112.65:38031)
I20260928 23:44:38.393419 31206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.65:57143
W20260928 23:44:38.506963 31397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:38.507220 31397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:38.507256 31397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:38.511396 31397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:38.511446 31397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:38.511526 31397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:38.516603 31397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:38.517818 31397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:38.519235 31397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:38.529826 31403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:38.529986 31405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.530055 31397 server_base.cc:1034] running on GCE node
W20260928 23:44:38.530078 31402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.530763 31397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:38.531399 31397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:38.532596 31397 hybrid_clock.cc:648] HybridClock initialized: now 1790639078532574 us; error 34 us; skew 500 ppm
I20260928 23:44:38.534672 31397 webserver.cc:460] Webserver started at http://127.21.112.66:36169/ using document root <none> and password file <none>
I20260928 23:44:38.535331 31397 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:38.535426 31397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:38.535669 31397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:38.538025 31397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "98df3e0cf7134e83a31bad809e00e146"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "9e8491b11b06f024509c686f15bb4cc8"
server_key_iv: "fc479b3eb351db2fe2a21171afdeb510"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.538623 31397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "98df3e0cf7134e83a31bad809e00e146"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "9e8491b11b06f024509c686f15bb4cc8"
server_key_iv: "fc479b3eb351db2fe2a21171afdeb510"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.542508 31397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:44:38.546108 31411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.547555 31397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:44:38.547686 31397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "98df3e0cf7134e83a31bad809e00e146"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "9e8491b11b06f024509c686f15bb4cc8"
server_key_iv: "fc479b3eb351db2fe2a21171afdeb510"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.547806 31397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:38.566030 31397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:38.566783 31397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:38.567031 31397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:38.567672 31397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:38.568665 31397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:38.568712 31397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.568753 31397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:38.568814 31397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.578945 31397 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:37067
I20260928 23:44:38.578986 31524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:37067 every 8 connection(s)
I20260928 23:44:38.579917 31397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:38.587431 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31397
I20260928 23:44:38.587581 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:38.587893 21953 external_mini_cluster.cc:1428] Setting key b4aebb9b312cda0e7ab642453f9166e2
I20260928 23:44:38.589987 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.581890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:37473 (local address 127.21.112.126:34599)
0928 23:44:38.582046 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:38.582050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:38.583146 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 23:44:38.584562 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:38.584565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:38.584610 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:38.584688 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:38.585716 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.586283 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.587572 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.587877 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.588002 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.589390 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:38.589394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:38.589396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:38.589410 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:38.589436 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:38.589448 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:38.589523 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:38.589771 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:44:38.589839 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:44:38.590390 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34599
--builtin_ntp_servers=127.21.112.84:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:44:38.590966 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.582341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34599 (local address 127.21.112.66:37473)
0928 23:44:38.583006 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:38.583049 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:38.584293 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:38.584858 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:38.584865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:38.584946 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:38.585512 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:38.585527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.586450 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.586453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:38.587425 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:38.587435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.588067 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.588074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:38.588274 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.588981 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:38.588983 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:38.589040 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:38.589124 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:38.589578 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:38.589588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:38.589594 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:44:38.589740 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 23:44:38.589987 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":101,"threads_started":1}
I20260928 23:44:38.592363 31525 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34599
I20260928 23:44:38.592628 31525 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:38.593160 31525 heartbeater.cc:511] Master 127.21.112.126:34599 requested a full tablet report, sending...
I20260928 23:44:38.594336 31206 ts_manager.cc:194] Registered new tserver with Master: 98df3e0cf7134e83a31bad809e00e146 (127.21.112.66:37067)
I20260928 23:44:38.595198 31206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.66:37473
W20260928 23:44:38.708050 31528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:38.708324 31528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:38.708362 31528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:38.712338 31528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:38.712397 31528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:38.712452 31528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:38.717124 31528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:34599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:38.718248 31528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:38.719610 31528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:44:38.730558 31528 server_base.cc:1034] running on GCE node
W20260928 23:44:38.730479 31533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:38.730479 31536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:38.730491 31534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.731258 31528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:38.731933 31528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:38.733144 31528 hybrid_clock.cc:648] HybridClock initialized: now 1790639078733124 us; error 34 us; skew 500 ppm
I20260928 23:44:38.735193 31528 webserver.cc:460] Webserver started at http://127.21.112.67:38613/ using document root <none> and password file <none>
I20260928 23:44:38.735881 31528 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:38.735980 31528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:38.736225 31528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:38.738610 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "97d96169143848339be4f8f7237a0f21"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "86d57aa6df111b0df22d54d338dc7ced"
server_key_iv: "47f9bba819ed754eef6220d91a42bbdf"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.739202 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "97d96169143848339be4f8f7237a0f21"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "86d57aa6df111b0df22d54d338dc7ced"
server_key_iv: "47f9bba819ed754eef6220d91a42bbdf"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.743296 31528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:38.746464 31542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.747938 31528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:38.748106 31528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "97d96169143848339be4f8f7237a0f21"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "86d57aa6df111b0df22d54d338dc7ced"
server_key_iv: "47f9bba819ed754eef6220d91a42bbdf"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.748243 31528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:38.763473 31528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:38.764258 31528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:38.764477 31528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:38.765110 31528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:38.766120 31528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:38.766192 31528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.766263 31528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:38.766315 31528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.775219 31528 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:39177
I20260928 23:44:38.775235 31655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:39177 every 8 connection(s)
I20260928 23:44:38.776216 31528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:38.778417 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31528
I20260928 23:44:38.778563 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:38.778859 21953 external_mini_cluster.cc:1428] Setting key acff508cf53b3127d8077ef912f656c7
I20260928 23:44:38.786106 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.778233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43923 (local address 127.21.112.126:34599)
0928 23:44:38.778456 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:38.778463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:38.779699 (+ 1236us) server_negotiation.cc:407] Connection header received
0928 23:44:38.781164 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:38.781168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:38.781221 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:38.781300 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:38.782287 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.782851 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.784072 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.784303 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.784419 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.785588 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:38.785591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:38.785594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:38.785608 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:38.785637 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:38.785645 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:38.785722 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:38.785895 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:44:38.785961 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 23:44:38.787003 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:38.778763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:34599 (local address 127.21.112.67:43923)
0928 23:44:38.779546 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:38.779600 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:38.780943 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:38.781479 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:38.781486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:38.781636 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:38.782137 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:38.782149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.782997 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.783003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:38.783938 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:38.783947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:38.784457 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:38.784461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:38.784634 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:38.785291 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:38.785293 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:38.785341 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:38.785408 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:38.785736 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:38.785746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:38.785759 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:44:38.785885 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 23:44:38.786206 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":162,"threads_started":1}
I20260928 23:44:38.788162 31656 heartbeater.cc:347] Connected to a master server at 127.21.112.126:34599
I20260928 23:44:38.788472 31656 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:38.788969 31656 heartbeater.cc:511] Master 127.21.112.126:34599 requested a full tablet report, sending...
I20260928 23:44:38.790112 31206 ts_manager.cc:194] Registered new tserver with Master: 97d96169143848339be4f8f7237a0f21 (127.21.112.67:39177)
I20260928 23:44:38.790772 31206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.67:43923
I20260928 23:44:38.792974 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:44:38.795964 31245 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:38.795023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.21.112.126:34599)
0928 23:44:38.795144 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:38.795147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:38.795366 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:44:38.795458 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:38.795460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:38.795501 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:38.795577 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:38.795793 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49204: BlockingRecv error: recv got EOF from 127.0.0.1:49204 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:44:38.796447 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31266
I20260928 23:44:38.802496 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31397
I20260928 23:44:38.807686 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31528
I20260928 23:44:38.813032 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31175
2026-09-28T23:44:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (904 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T23:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:38Z Disabled control of system clock
I20260928 23:44:38.842499 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:33491
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:33491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:44:38.956151 31667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:38.956441 31667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:38.956478 31667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:38.961158 31667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:38.961222 31667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:38.961237 31667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:38.961256 31667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:38.961272 31667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:38.966259 31667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:33491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:33491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:38.967368 31667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:38.968825 31667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:38.977674 31673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:38.977674 31672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.977815 31667 server_base.cc:1034] running on GCE node
W20260928 23:44:38.977685 31675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:38.978659 31667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:38.979770 31667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:38.980945 31667 hybrid_clock.cc:648] HybridClock initialized: now 1790639078980926 us; error 34 us; skew 500 ppm
I20260928 23:44:38.982865 31667 webserver.cc:460] Webserver started at http://127.21.112.126:35531/ using document root <none> and password file <none>
I20260928 23:44:38.983420 31667 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:38.983481 31667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:38.983649 31667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:38.985810 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "163b1ae75cc748489564815807507eec"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "1325a3844aaf1faf0bf0a392305be4e3"
server_key_iv: "79f087424a2da5c1bc24edf49099dfdc"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.986326 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "163b1ae75cc748489564815807507eec"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "1325a3844aaf1faf0bf0a392305be4e3"
server_key_iv: "79f087424a2da5c1bc24edf49099dfdc"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.990144 31667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:44:38.992575 31681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:38.993916 31667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:38.994047 31667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "163b1ae75cc748489564815807507eec"
format_stamp: "Formatted at 2026-09-28 23:44:38 on dist-test-slave-c3s8"
server_key: "1325a3844aaf1faf0bf0a392305be4e3"
server_key_iv: "79f087424a2da5c1bc24edf49099dfdc"
server_key_version: "encryptionkey@0"
I20260928 23:44:38.994166 31667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:38.998937 31667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:38.999606 31667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:38.999804 31667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:39.006980 31667 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:33491
I20260928 23:44:39.007000 31733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:33491 every 8 connection(s)
I20260928 23:44:39.007997 31667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:39.011615 31734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:39.017251 31734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec: Bootstrap starting.
I20260928 23:44:39.018247 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31667
I20260928 23:44:39.018368 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:39.018685 21953 external_mini_cluster.cc:1428] Setting key 390f89ae6085358521da89b81a71cec9
I20260928 23:44:39.020315 31734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:39.021076 31734 log.cc:824] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:39.023468 31734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec: No bootstrap required, opened a new log
I20260928 23:44:39.024361 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.019912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.21.112.126:33491)
0928 23:44:39.020642 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:39.020659 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:39.020694 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:44:39.021427 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:39.021455 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:39.021850 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:39.022179 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:39.023002 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:39.023007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:39.023018 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:39.023029 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:39.023095 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:39.023112 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:39.023207 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:39.023364 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 23:44:39.023652 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":139,"threads_started":1}
I20260928 23:44:39.027638 31734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b1ae75cc748489564815807507eec" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33491 } }
I20260928 23:44:39.027885 31734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:39.027976 31734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163b1ae75cc748489564815807507eec, State: Initialized, Role: FOLLOWER
I20260928 23:44:39.028469 31734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b1ae75cc748489564815807507eec" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33491 } }
I20260928 23:44:39.028620 31734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:39.028704 31734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:39.028821 31734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:39.029927 31734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b1ae75cc748489564815807507eec" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33491 } }
I20260928 23:44:39.030273 31734 leader_election.cc:302] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 163b1ae75cc748489564815807507eec; no voters:
I20260928 23:44:39.030586 31734 leader_election.cc:288] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:39.030736 31739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:39.031126 31739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [term 1 LEADER]: Becoming Leader. State: Replica: 163b1ae75cc748489564815807507eec, State: Running, Role: LEADER
I20260928 23:44:39.031476 31739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b1ae75cc748489564815807507eec" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33491 } }
I20260928 23:44:39.031967 31734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:39.033430 31741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 163b1ae75cc748489564815807507eec. Latest consensus state: current_term: 1 leader_uuid: "163b1ae75cc748489564815807507eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b1ae75cc748489564815807507eec" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33491 } } }
I20260928 23:44:39.033454 31740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "163b1ae75cc748489564815807507eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b1ae75cc748489564815807507eec" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 33491 } } }
I20260928 23:44:39.033612 31740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:39.033878 31741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:39.034112 31748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:39.037362 31748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:39.042845 31748 catalog_manager.cc:1347] Generated new cluster ID: 27c963f937b14a5fac2fb84ddf6ca669
I20260928 23:44:39.042932 31748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:39.055604 31748 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:39.055971 31748 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:39.064141 31748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 163b1ae75cc748489564815807507eec: Generated new TSK 0
I20260928 23:44:39.064859 31748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:44:39.074182 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33491
--builtin_ntp_servers=127.21.112.84:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:44:39.189571 31758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:39.189843 31758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:39.189885 31758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:39.194543 31758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:39.194618 31758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:39.194805 31758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:39.200524 31758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:39.201813 31758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:39.203307 31758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:39.213425 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:39.213425 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:39.213507 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:39.213713 31758 server_base.cc:1034] running on GCE node
I20260928 23:44:39.214159 31758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:39.214864 31758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:39.216071 31758 hybrid_clock.cc:648] HybridClock initialized: now 1790639079216056 us; error 27 us; skew 500 ppm
I20260928 23:44:39.218084 31758 webserver.cc:460] Webserver started at http://127.21.112.65:46409/ using document root <none> and password file <none>
I20260928 23:44:39.218672 31758 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:39.218767 31758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:39.219005 31758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:39.221115 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "aebf5265458a40a39db87f164ea7bf44"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "454fcda66166611f9b0b683dc14db5b8"
server_key_iv: "2ca95d394ce762fd0167eb71bca2793c"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.221735 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "aebf5265458a40a39db87f164ea7bf44"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "454fcda66166611f9b0b683dc14db5b8"
server_key_iv: "2ca95d394ce762fd0167eb71bca2793c"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.225375 31758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:44:39.227999 31772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.229225 31758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:39.229365 31758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "aebf5265458a40a39db87f164ea7bf44"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "454fcda66166611f9b0b683dc14db5b8"
server_key_iv: "2ca95d394ce762fd0167eb71bca2793c"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.229485 31758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:39.234776 31758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:39.235417 31758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:39.235610 31758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:39.236163 31758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:39.237102 31758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:39.237174 31758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.237242 31758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:39.237299 31758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.246757 31758 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:36573
I20260928 23:44:39.246783 31885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:36573 every 8 connection(s)
I20260928 23:44:39.247702 31758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:39.250913 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31758
I20260928 23:44:39.251062 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:39.251389 21953 external_mini_cluster.cc:1428] Setting key 6f65e78c4b4c4b35b1214217eb679f92
I20260928 23:44:39.253610 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33491
--builtin_ntp_servers=127.21.112.84:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:44:39.254961 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.249753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:57887 (local address 127.21.112.126:33491)
0928 23:44:39.249919 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:39.249925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:39.251169 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 23:44:39.252991 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:39.252994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:39.253028 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:39.253084 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:39.254501 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:39.254504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:39.254506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:39.254518 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:39.254538 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:39.254546 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:39.254622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:39.254793 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:44:39.254829 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:44:39.255935 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.250192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33491 (local address 127.21.112.65:57887)
0928 23:44:39.250989 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:39.251054 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:39.252765 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:39.253306 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:39.253317 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:39.253452 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:39.254179 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:39.254183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:39.254244 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:39.254339 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:39.254635 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:39.254647 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:39.254656 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:44:39.254837 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 23:44:39.255201 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260928 23:44:39.257268 31886 heartbeater.cc:347] Connected to a master server at 127.21.112.126:33491
I20260928 23:44:39.257568 31886 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:39.258064 31886 heartbeater.cc:511] Master 127.21.112.126:33491 requested a full tablet report, sending...
I20260928 23:44:39.259933 31698 ts_manager.cc:194] Registered new tserver with Master: aebf5265458a40a39db87f164ea7bf44 (127.21.112.65:36573)
W20260928 23:44:39.375960 31889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:39.376257 31889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:39.376304 31889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:39.380806 31889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:39.380870 31889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:39.380930 31889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:39.386186 31889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:39.387362 31889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:39.388813 31889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:39.399050 31897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:39.399010 31895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:39.399156 31894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:39.399297 31889 server_base.cc:1034] running on GCE node
I20260928 23:44:39.399739 31889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:39.400468 31889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:39.401612 31889 hybrid_clock.cc:648] HybridClock initialized: now 1790639079401562 us; error 61 us; skew 500 ppm
I20260928 23:44:39.403453 31889 webserver.cc:460] Webserver started at http://127.21.112.66:43551/ using document root <none> and password file <none>
I20260928 23:44:39.404067 31889 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:39.404162 31889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:39.404376 31889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:39.406550 31889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "aa379e249112406d822956ad7dd6655a"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "7e13f91f921d6cd30591f5973fa3ed5b"
server_key_iv: "550d01067053b37306b3e284865cbeaa"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.407127 31889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "aa379e249112406d822956ad7dd6655a"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "7e13f91f921d6cd30591f5973fa3ed5b"
server_key_iv: "550d01067053b37306b3e284865cbeaa"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.410840 31889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:44:39.413355 31903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.414646 31889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:39.414788 31889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "aa379e249112406d822956ad7dd6655a"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "7e13f91f921d6cd30591f5973fa3ed5b"
server_key_iv: "550d01067053b37306b3e284865cbeaa"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.414907 31889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:39.419605 31889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:39.420215 31889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:39.420435 31889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:39.421001 31889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:39.421959 31889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:39.422029 31889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.422094 31889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:39.422143 31889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.431509 31889 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:38187
I20260928 23:44:39.431543 32016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:38187 every 8 connection(s)
I20260928 23:44:39.432525 31889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:39.439209 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.434621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:59177 (local address 127.21.112.126:33491)
0928 23:44:39.434774 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:39.434777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:39.435873 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 23:44:39.437286 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:39.437289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:39.437336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:39.437462 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:39.438682 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:39.438685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:39.438688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:39.438701 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:39.438724 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:39.438759 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:39.438858 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:39.439021 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 23:44:39.439068 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:44:39.440132 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.435036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33491 (local address 127.21.112.66:59177)
0928 23:44:39.435729 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:39.435769 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:39.437070 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:39.437688 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:39.437696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:39.437806 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:39.438413 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:39.438415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:39.438457 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:39.438524 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:39.438853 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:39.438863 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:39.438871 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:44:39.439061 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 23:44:39.439302 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260928 23:44:39.440986 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 31889
I20260928 23:44:39.441130 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:39.441313 32017 heartbeater.cc:347] Connected to a master server at 127.21.112.126:33491
I20260928 23:44:39.441421 21953 external_mini_cluster.cc:1428] Setting key 5439d335b83746f92fbbdfbd1589c771
I20260928 23:44:39.441615 32017 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:39.442070 32017 heartbeater.cc:511] Master 127.21.112.126:33491 requested a full tablet report, sending...
I20260928 23:44:39.443068 31698 ts_manager.cc:194] Registered new tserver with Master: aa379e249112406d822956ad7dd6655a (127.21.112.66:38187)
I20260928 23:44:39.443400 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33491
--builtin_ntp_servers=127.21.112.84:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:44:39.555425 32020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:39.555706 32020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:39.555747 32020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:39.559938 32020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:39.560001 32020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:39.560058 32020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:39.565186 32020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:33491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:39.566406 32020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:39.568234 32020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:39.578145 32025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:39.578146 32028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:39.578315 32020 server_base.cc:1034] running on GCE node
W20260928 23:44:39.578250 32026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:39.578838 32020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:39.579484 32020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:39.580673 32020 hybrid_clock.cc:648] HybridClock initialized: now 1790639079580657 us; error 32 us; skew 500 ppm
I20260928 23:44:39.582551 32020 webserver.cc:460] Webserver started at http://127.21.112.67:42941/ using document root <none> and password file <none>
I20260928 23:44:39.583129 32020 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:39.583221 32020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:39.583436 32020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:39.585429 32020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "540ef925c7264ae6a5a9fef50442e6d5"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "4d387ace9ff6c964b008a7ad27f20da3"
server_key_iv: "58e92ab5bb132f16c85200508f75d570"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.586053 32020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "540ef925c7264ae6a5a9fef50442e6d5"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "4d387ace9ff6c964b008a7ad27f20da3"
server_key_iv: "58e92ab5bb132f16c85200508f75d570"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.589798 32020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:44:39.592316 32034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.593734 32020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:39.593868 32020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "540ef925c7264ae6a5a9fef50442e6d5"
format_stamp: "Formatted at 2026-09-28 23:44:39 on dist-test-slave-c3s8"
server_key: "4d387ace9ff6c964b008a7ad27f20da3"
server_key_iv: "58e92ab5bb132f16c85200508f75d570"
server_key_version: "encryptionkey@0"
I20260928 23:44:39.593992 32020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:39.598832 32020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:39.599555 32020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:39.599743 32020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:39.600301 32020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:39.601277 32020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:39.601349 32020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.601420 32020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:39.601478 32020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:39.610284 32020 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:35681
I20260928 23:44:39.610304 32147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:35681 every 8 connection(s)
I20260928 23:44:39.611227 32020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:39.617403 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.613204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:49751 (local address 127.21.112.126:33491)
0928 23:44:39.613365 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:39.613369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:39.614579 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 23:44:39.615872 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:39.615875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:39.615922 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:39.616001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:39.617027 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:39.617030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:39.617032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:44:39.617045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:39.617065 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:44:39.617073 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:39.617150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:39.617251 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:44:39.617289 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:44:39.618429 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:39.613703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:33491 (local address 127.21.112.67:49751)
0928 23:44:39.614388 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:39.614430 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:39.615661 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:39.616143 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:39.616150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:39.616243 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:39.616767 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:44:39.616769 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:39.616813 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:44:39.616885 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:39.617138 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:39.617142 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:39.617146 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:44:39.617269 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 23:44:39.617514 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260928 23:44:39.619683 32148 heartbeater.cc:347] Connected to a master server at 127.21.112.126:33491
I20260928 23:44:39.619712 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 32020
I20260928 23:44:39.619848 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:39.619956 32148 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:39.620175 21953 external_mini_cluster.cc:1428] Setting key 671250e4b5dce34e9a228d870dd82789
I20260928 23:44:39.620428 32148 heartbeater.cc:511] Master 127.21.112.126:33491 requested a full tablet report, sending...
I20260928 23:44:39.621575 31698 ts_manager.cc:194] Registered new tserver with Master: 540ef925c7264ae6a5a9fef50442e6d5 (127.21.112.67:35681)
I20260928 23:44:39.622503 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:44:39.625157 31737 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:44:39.624383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.21.112.126:33491)
0928 23:44:39.624527 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:39.624530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:39.624693 (+ 163us) server_negotiation.cc:407] Connection header received
0928 23:44:39.624773 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:39.624775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:39.624803 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:39.624864 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:39.625025 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34538: BlockingRecv error: recv got EOF from 127.0.0.1:34538 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:44:39.625648 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31758
I20260928 23:44:39.631873 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31889
I20260928 23:44:39.637321 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 32020
I20260928 23:44:39.642649 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 31667
2026-09-28T23:44:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (829 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:39 dist-test-slave-c3s8 krb5kdc[32158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:39 dist-test-slave-c3s8 krb5kdc[32158](info): set up 2 sockets
Sep 28 23:44:39 dist-test-slave-c3s8 krb5kdc[32158](info): commencing operation
krb5kdc: starting...
W20260928 23:44:40.755051 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:40 dist-test-slave-c3s8 krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:44:40.889690 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:44659
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:44659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:41.010910 32171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:41.011242 32171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:41.011287 32171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:41.015648 32171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:44:41.015723 32171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:41.015743 32171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:41.015760 32171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:44:41.015776 32171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:44:41.021299 32171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:44659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:44659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:41.022485 32171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:41.023484 32171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:41.032521 32177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:41.032585 32176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.032648 32171 server_base.cc:1034] running on GCE node
W20260928 23:44:41.032586 32179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.033363 32171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:41.034590 32171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:41.035760 32171 hybrid_clock.cc:648] HybridClock initialized: now 1790639081035731 us; error 51 us; skew 500 ppm
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:41.046754 32171 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:44:41.048005 32171 webserver.cc:460] Webserver started at http://127.21.112.126:41401/ using document root <none> and password file <none>
I20260928 23:44:41.048732 32171 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:41.048914 32171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:41.049260 32171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:41.052359 32171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "c140be21d8e842e0844eb189165f4ec5"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "cbf2499701ad62ebe68ce51bd47ffa78"
server_key_iv: "3dacaa6c3b7a0096f7c3262dcdd832ea"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.053154 32171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "c140be21d8e842e0844eb189165f4ec5"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "cbf2499701ad62ebe68ce51bd47ffa78"
server_key_iv: "3dacaa6c3b7a0096f7c3262dcdd832ea"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.057770 32171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 23:44:41.060488 32186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.062016 32171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:44:41.062176 32171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "c140be21d8e842e0844eb189165f4ec5"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "cbf2499701ad62ebe68ce51bd47ffa78"
server_key_iv: "3dacaa6c3b7a0096f7c3262dcdd832ea"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.062307 32171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:41.079242 32171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:41.082670 32171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:41.082887 32171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:41.090121 32171 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:44659
I20260928 23:44:41.090440 32238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:44659 every 8 connection(s)
I20260928 23:44:41.091143 32171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:44:41.094123 32239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:41.097205 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 32171
I20260928 23:44:41.097338 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:44:41.097647 21953 external_mini_cluster.cc:1428] Setting key e1d863bd2b8748c1cca6cf31fe55d052
I20260928 23:44:41.100297 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5: Bootstrap starting.
I20260928 23:44:41.102954 32239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:41.103724 32239 log.cc:824] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:41.105955 32239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5: No bootstrap required, opened a new log
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:41.109366 32239 raft_consensus.cc:348] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c140be21d8e842e0844eb189165f4ec5" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44659 } }
I20260928 23:44:41.109691 32239 raft_consensus.cc:374] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:41.109786 32239 raft_consensus.cc:729] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c140be21d8e842e0844eb189165f4ec5, State: Initialized, Role: FOLLOWER
I20260928 23:44:41.110311 32239 consensus_queue.cc:260] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c140be21d8e842e0844eb189165f4ec5" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44659 } }
I20260928 23:44:41.110469 32239 raft_consensus.cc:388] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:44:41.110538 32239 raft_consensus.cc:482] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:44:41.110646 32239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:41.111595 32239 raft_consensus.cc:504] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c140be21d8e842e0844eb189165f4ec5" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44659 } }
I20260928 23:44:41.111965 32239 leader_election.cc:302] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c140be21d8e842e0844eb189165f4ec5; no voters:
I20260928 23:44:41.112334 32239 leader_election.cc:288] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:44:41.112411 32244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:41.112622 32244 raft_consensus.cc:686] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [term 1 LEADER]: Becoming Leader. State: Replica: c140be21d8e842e0844eb189165f4ec5, State: Running, Role: LEADER
I20260928 23:44:41.112946 32244 consensus_queue.cc:237] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c140be21d8e842e0844eb189165f4ec5" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44659 } }
I20260928 23:44:41.113730 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.099114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.21.112.126:44659)
0928 23:44:41.099570 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.099594 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.099631 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:44:41.100241 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.100258 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.100656 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.100946 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:41.101904 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.102918 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.103867 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.104616 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.104796 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.107375 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:41.107381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:41.107401 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:41.107429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:41.110215 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.110747 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.110754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.110760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.110891 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.111225 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.111229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.111231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.111656 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:41.111880 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.112192 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.112709 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":162,"threads_started":1}
I20260928 23:44:41.113754 32239 sys_catalog.cc:564] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:44:41.114605 32246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c140be21d8e842e0844eb189165f4ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c140be21d8e842e0844eb189165f4ec5" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44659 } } }
I20260928 23:44:41.114926 32245 sys_catalog.cc:455] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c140be21d8e842e0844eb189165f4ec5. Latest consensus state: current_term: 1 leader_uuid: "c140be21d8e842e0844eb189165f4ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c140be21d8e842e0844eb189165f4ec5" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 44659 } } }
I20260928 23:44:41.115087 32245 sys_catalog.cc:458] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:41.115048 32246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:44:41.116094 32253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:44:41.119218 32253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:44:41.124769 32253 catalog_manager.cc:1347] Generated new cluster ID: 3f8252176e0b43c5a0e845f8edfefe66
I20260928 23:44:41.124858 32253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:44:41.145478 32253 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:44:41.146553 32253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:44:41.163002 32253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5: Generated new TSK 0
I20260928 23:44:41.163861 32253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:44:41.209785 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44659
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:41.344271 32265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:41.344712 32265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:41.344763 32265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:41.348879 32265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:41.348945 32265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:41.349011 32265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:44:41.354478 32265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:41.355644 32265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:41.356714 32265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:41.366700 32271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.366808 32265 server_base.cc:1034] running on GCE node
W20260928 23:44:41.366717 32270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:41.366717 32273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.367528 32265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:41.368188 32265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:41.369386 32265 hybrid_clock.cc:648] HybridClock initialized: now 1790639081369346 us; error 41 us; skew 500 ppm
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:41.372337 32265 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:44:41.373266 32265 webserver.cc:460] Webserver started at http://127.21.112.65:46099/ using document root <none> and password file <none>
I20260928 23:44:41.373966 32265 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:41.374056 32265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:41.374296 32265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:41.376348 32265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "4be9c995da1541b2c89a6d9ed04759cd"
server_key_iv: "02b523277b2a8ba7d8fb92d4a440fae6"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.376962 32265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "4be9c995da1541b2c89a6d9ed04759cd"
server_key_iv: "02b523277b2a8ba7d8fb92d4a440fae6"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.380911 32265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:44:41.383407 32280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.384621 32265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:44:41.384760 32265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "4be9c995da1541b2c89a6d9ed04759cd"
server_key_iv: "02b523277b2a8ba7d8fb92d4a440fae6"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.384877 32265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:41.399894 32265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:41.403313 32265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:41.403525 32265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:41.404136 32265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:41.404956 32265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:41.405030 32265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.405115 32265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:41.405165 32265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.414410 32265 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:36363
I20260928 23:44:41.414465 32393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:36363 every 8 connection(s)
I20260928 23:44:41.415504 32265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:44:41.417639 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 32265
I20260928 23:44:41.417760 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:44:41.418072 21953 external_mini_cluster.cc:1428] Setting key 61c3e3bff03f6b98e2b047b4fa6d73e7
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:44:41.431785 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.417388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:34843 (local address 127.21.112.126:44659)
0928 23:44:41.417561 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.417568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.418936 (+ 1368us) server_negotiation.cc:407] Connection header received
0928 23:44:41.420438 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.420444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.420509 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.420604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:41.422222 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.422815 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.424659 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.424892 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.424993 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.427955 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:41.427959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:41.427962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:41.427998 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:41.429620 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.430191 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.430195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.430196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.430285 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.430648 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.430651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.430653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.430849 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:41.430959 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.431489 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.431613 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:44:41.432734 32396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.417957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44659 (local address 127.21.112.65:34843)
0928 23:44:41.418565 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.418626 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.420201 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.420834 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.420842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.421299 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:41.422050 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.422072 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.422954 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.422961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.424506 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.424519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.425076 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.425084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.425319 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.425993 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:41.426022 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:41.427762 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:41.429742 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:41.429749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:41.429767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:41.430074 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:41.430380 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:41.430384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:41.430387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:41.430535 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:41.430961 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:41.430971 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:41.431232 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.431424 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.431766 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":174,"threads_started":1}
I20260928 23:44:41.434386 32394 heartbeater.cc:347] Connected to a master server at 127.21.112.126:44659
I20260928 23:44:41.434679 32394 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:41.435336 32394 heartbeater.cc:511] Master 127.21.112.126:44659 requested a full tablet report, sending...
I20260928 23:44:41.437057 32203 ts_manager.cc:194] Registered new tserver with Master: 4e71cc9cb9bf46cc90b89e5ba4e907bb (127.21.112.65:36363)
I20260928 23:44:41.438357 32203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:34843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:44:41.449306 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44659
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:41.562597 32399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:41.562875 32399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:41.562918 32399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:41.567147 32399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:41.567212 32399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:41.567308 32399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:44:41.572849 32399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:41.574117 32399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:41.575255 32399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:41.585244 32407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:41.585239 32405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:41.585244 32404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.585887 32399 server_base.cc:1034] running on GCE node
I20260928 23:44:41.586431 32399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:41.587157 32399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:41.588316 32399 hybrid_clock.cc:648] HybridClock initialized: now 1790639081588284 us; error 45 us; skew 500 ppm
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:41.591185 32399 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:44:41.592145 32399 webserver.cc:460] Webserver started at http://127.21.112.66:38763/ using document root <none> and password file <none>
I20260928 23:44:41.592743 32399 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:41.592804 32399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:41.593041 32399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:41.595183 32399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "ccec7bacd1be4f75aaa6bf2316605df7"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "a831c0d623772a895a3e102933d16b68"
server_key_iv: "28fb1afda19355ed6c9b92ec7bd37a27"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.595781 32399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "ccec7bacd1be4f75aaa6bf2316605df7"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "a831c0d623772a895a3e102933d16b68"
server_key_iv: "28fb1afda19355ed6c9b92ec7bd37a27"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.599550 32399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:41.602087 32414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.603353 32399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:44:41.603499 32399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "ccec7bacd1be4f75aaa6bf2316605df7"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "a831c0d623772a895a3e102933d16b68"
server_key_iv: "28fb1afda19355ed6c9b92ec7bd37a27"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.603615 32399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:41.623133 32399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:41.626655 32399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:41.626878 32399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:41.627507 32399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:41.628350 32399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:41.628424 32399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.628502 32399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:41.628552 32399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.638113 32399 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:41571
I20260928 23:44:41.638134 32527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:41571 every 8 connection(s)
I20260928 23:44:41.639134 32399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:44:41.646278 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 32399
I20260928 23:44:41.646431 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:44:41.646744 21953 external_mini_cluster.cc:1428] Setting key 821beafc095d00a370143a0319fb4142
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:41.654928 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.641285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:51777 (local address 127.21.112.126:44659)
0928 23:44:41.641495 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.641499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.642432 (+ 933us) server_negotiation.cc:407] Connection header received
0928 23:44:41.643924 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.643928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.643985 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.644097 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:41.645452 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.646077 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.647498 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.647733 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.647854 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.650781 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:41.650787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:41.650791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:41.650841 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:41.652594 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.653216 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.653220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.653222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.653327 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.653754 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.653757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.653759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.653968 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:41.654073 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.654625 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.654763 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 23:44:41.655797 32530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.641735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44659 (local address 127.21.112.66:51777)
0928 23:44:41.642248 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.642303 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.643691 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.644287 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.644294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.644667 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:41.645300 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.645315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.646204 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.646210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.647377 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.647387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.647894 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.647898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.648116 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.648822 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:41.648839 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:41.650564 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:41.652748 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:41.652754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:41.652775 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:41.653094 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:41.653420 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:41.653424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:41.653426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:41.653648 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:41.654073 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:41.654081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:41.654368 (+ 287us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.654568 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.654939 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:44:41.657091 32528 heartbeater.cc:347] Connected to a master server at 127.21.112.126:44659
I20260928 23:44:41.657378 32528 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:41.657943 32528 heartbeater.cc:511] Master 127.21.112.126:44659 requested a full tablet report, sending...
I20260928 23:44:41.659160 32203 ts_manager.cc:194] Registered new tserver with Master: ccec7bacd1be4f75aaa6bf2316605df7 (127.21.112.66:41571)
I20260928 23:44:41.659857 32203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:51777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:44:41.678118 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44659
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:44:41.798393 32533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:44:41.798710 32533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:44:41.798782 32533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:44:41.802942 32533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:44:41.803042 32533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:44:41.803133 32533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:44:41.808758 32533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:44659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:44:41.810000 32533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:44:41.811072 32533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:44:41.821213 32541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:44:41.821208 32538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.821312 32533 server_base.cc:1034] running on GCE node
W20260928 23:44:41.821204 32539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:44:41.822026 32533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:44:41.822719 32533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:44:41.823913 32533 hybrid_clock.cc:648] HybridClock initialized: now 1790639081823889 us; error 35 us; skew 500 ppm
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:44:41.826706 32533 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:44:41.827613 32533 webserver.cc:460] Webserver started at http://127.21.112.67:44593/ using document root <none> and password file <none>
I20260928 23:44:41.828186 32533 fs_manager.cc:359] Metadata directory not provided
I20260928 23:44:41.828279 32533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:44:41.828526 32533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:44:41.830857 32533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "db1d870dc0114f15993352ac1ac46a18"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "7ba4a0d03c399d5d6052be2d88721331"
server_key_iv: "d98b830206ecd8b43fdcd1207fce653d"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.831454 32533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "db1d870dc0114f15993352ac1ac46a18"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "7ba4a0d03c399d5d6052be2d88721331"
server_key_iv: "d98b830206ecd8b43fdcd1207fce653d"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.835377 32533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:44:41.837929 32548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.839205 32533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:44:41.839353 32533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "db1d870dc0114f15993352ac1ac46a18"
format_stamp: "Formatted at 2026-09-28 23:44:41 on dist-test-slave-c3s8"
server_key: "7ba4a0d03c399d5d6052be2d88721331"
server_key_iv: "d98b830206ecd8b43fdcd1207fce653d"
server_key_version: "encryptionkey@0"
I20260928 23:44:41.839489 32533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:44:41.850693 32533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:44:41.853443 32533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:44:41.853662 32533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:44:41.854243 32533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:44:41.855062 32533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:44:41.855131 32533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.855201 32533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:44:41.855253 32533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:44:41.864539 32533 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:41711
I20260928 23:44:41.864565 32661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:41711 every 8 connection(s)
I20260928 23:44:41.865453 32533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:44:41.875234 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 32533
I20260928 23:44:41.875347 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:44:41.875641 21953 external_mini_cluster.cc:1428] Setting key 518e8afa1613b7774a789407a258391b
Sep 28 23:44:41 dist-test-slave-c3s8 krb5kdc[32158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639081, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:44:41.880347 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.867485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43931 (local address 127.21.112.126:44659)
0928 23:44:41.867671 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.867676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.868643 (+ 967us) server_negotiation.cc:407] Connection header received
0928 23:44:41.870033 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.870036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.870096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.870185 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:41.871396 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.871975 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.873133 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.873441 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.873610 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.876503 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:41.876513 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:41.876516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:41.876542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:41.878234 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.878765 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.878770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.878772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.878856 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.879219 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.879224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.879225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.879421 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:41.879534 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.880059 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.880187 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:44:41.881320 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.867987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:44659 (local address 127.21.112.67:43931)
0928 23:44:41.868458 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.868524 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.869819 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.870357 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.870364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.870751 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:44:41.871229 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.871243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.872102 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.872104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.872999 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.873007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.873652 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.873655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.873801 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.874425 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:44:41.874442 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:44:41.876254 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:44:41.878354 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:41.878361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:41.878379 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:41.878655 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:41.878958 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:44:41.878961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:44:41.878963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:44:41.879115 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:44:41.879542 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:44:41.879551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:44:41.879809 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.880019 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.880367 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260928 23:44:41.882609 32662 heartbeater.cc:347] Connected to a master server at 127.21.112.126:44659
I20260928 23:44:41.882915 32662 heartbeater.cc:464] Registering TS with master...
I20260928 23:44:41.883471 32662 heartbeater.cc:511] Master 127.21.112.126:44659 requested a full tablet report, sending...
I20260928 23:44:41.884513 32203 ts_manager.cc:194] Registered new tserver with Master: db1d870dc0114f15993352ac1ac46a18 (127.21.112.67:41711)
I20260928 23:44:41.885092 32203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:43931
I20260928 23:44:41.890123 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:44:41.899861 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.892148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.21.112.126:44659)
0928 23:44:41.892286 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.892289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.892496 (+ 207us) server_negotiation.cc:407] Connection header received
0928 23:44:41.892594 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.892596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.892635 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.892702 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:44:41.893586 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.894118 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.895035 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.895249 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.895330 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.896853 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:44:41.896856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:44:41.896858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:44:41.896902 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:44:41.898311 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.898743 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.898745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.898747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.898825 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:44:41.899071 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:44:41.899073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:44:41.899074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:44:41.899295 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:44:41.899401 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.899639 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.899738 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:44:41.902853 32203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:44:41.906097 32203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:44:41.921706 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.915325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:36363 (local address 127.0.0.1:49774)
0928 23:44:41.916327 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.916388 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.916528 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.917207 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.917213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.917240 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:41.917432 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.917441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.918629 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.918635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.919875 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.919883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.921092 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.921095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.921210 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.921256 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.921375 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.921504 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"spinlock_wait_cycles":384,"thread_start_us":92,"threads_started":1}
I20260928 23:44:41.921732 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.915726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.21.112.65:36363)
0928 23:44:41.916128 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.916145 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.916690 (+ 545us) server_negotiation.cc:407] Connection header received
0928 23:44:41.916795 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.916802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.917041 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.917167 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:41.917580 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.918493 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.920011 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.920964 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.921080 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.921240 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.921423 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.921562 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260928 23:44:41.922247 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.915927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:41711 (local address 127.0.0.1:44232)
0928 23:44:41.916796 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.916809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.916882 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.917438 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.917440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.917456 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:41.917601 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.917609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.919181 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.919186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.920561 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.920573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.921830 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.921835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.921924 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.921957 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.922027 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.922098 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":26,"thread_start_us":125,"threads_started":1}
I20260928 23:44:41.922689 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.916383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.21.112.67:41711)
0928 23:44:41.916965 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.916975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.917000 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:44:41.917069 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.917073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.917284 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.917453 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:41.917737 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.919004 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.920704 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.921488 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.921627 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.921762 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.922427 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.922543 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":118,"threads_started":1}
I20260928 23:44:41.923019 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.915375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:41571 (local address 127.0.0.1:54886)
0928 23:44:41.916327 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.916387 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.916528 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.918315 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.918318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.918336 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:41.918417 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.918422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.919869 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.919872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.920723 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.920733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.922564 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.922569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.922701 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.922735 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.922798 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.922884 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":24,"spinlock_wait_cycles":6656,"thread_start_us":144,"threads_started":1}
I20260928 23:44:41.923637 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.915397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.21.112.66:41571)
0928 23:44:41.917857 (+ 2460us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.917869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.917892 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:44:41.917971 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.917976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.918155 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.918313 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:41.918523 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.919708 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.920866 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.922247 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.923282 (+ 1035us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.923406 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.923446 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.923521 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2335,"thread_start_us":153,"threads_started":1}
I20260928 23:44:41.925273 32328 tablet_service.cc:1467] Processing CreateTablet for tablet 4412a0aa512b40718ab92d0fb830225e (DEFAULT_TABLE table=test-table [id=52b68f38ffd74abfb41dc13ff028dc27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:41.926554 32328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4412a0aa512b40718ab92d0fb830225e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:41.926586 32596 tablet_service.cc:1467] Processing CreateTablet for tablet 4412a0aa512b40718ab92d0fb830225e (DEFAULT_TABLE table=test-table [id=52b68f38ffd74abfb41dc13ff028dc27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:41.926571 32462 tablet_service.cc:1467] Processing CreateTablet for tablet 4412a0aa512b40718ab92d0fb830225e (DEFAULT_TABLE table=test-table [id=52b68f38ffd74abfb41dc13ff028dc27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:44:41.927613 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4412a0aa512b40718ab92d0fb830225e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:41.927711 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4412a0aa512b40718ab92d0fb830225e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:44:41.933668 32679 tablet_bootstrap.cc:492] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Bootstrap starting.
I20260928 23:44:41.935753 32679 tablet_bootstrap.cc:654] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:41.936039 32681 tablet_bootstrap.cc:492] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: Bootstrap starting.
I20260928 23:44:41.937222 32679 log.cc:824] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:41.938292 32681 tablet_bootstrap.cc:654] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:41.939042 32681 log.cc:824] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:41.939282 32679 tablet_bootstrap.cc:492] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: No bootstrap required, opened a new log
I20260928 23:44:41.939484 32679 ts_tablet_manager.cc:1397] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260928 23:44:41.939877 32680 tablet_bootstrap.cc:492] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: Bootstrap starting.
I20260928 23:44:41.941206 32681 tablet_bootstrap.cc:492] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: No bootstrap required, opened a new log
I20260928 23:44:41.941406 32681 ts_tablet_manager.cc:1397] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:44:41.942348 32680 tablet_bootstrap.cc:654] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: Neither blocks nor log segments found. Creating new log.
I20260928 23:44:41.943184 32680 log.cc:824] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: Log is configured to *not* fsync() on all Append() calls
I20260928 23:44:41.943637 32679 raft_consensus.cc:348] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.943950 32679 raft_consensus.cc:374] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:41.944024 32679 raft_consensus.cc:729] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1d870dc0114f15993352ac1ac46a18, State: Initialized, Role: FOLLOWER
I20260928 23:44:41.944624 32679 consensus_queue.cc:260] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.945389 32662 heartbeater.cc:503] Master 127.21.112.126:44659 was elected leader, sending a full tablet report...
I20260928 23:44:41.945853 32679 ts_tablet_manager.cc:1428] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 23:44:41.946110 32681 raft_consensus.cc:348] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.946324 32681 raft_consensus.cc:374] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:41.946359 32681 raft_consensus.cc:729] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e71cc9cb9bf46cc90b89e5ba4e907bb, State: Initialized, Role: FOLLOWER
I20260928 23:44:41.946780 32680 tablet_bootstrap.cc:492] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: No bootstrap required, opened a new log
I20260928 23:44:41.946772 32681 consensus_queue.cc:260] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.947009 32680 ts_tablet_manager.cc:1397] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 23:44:41.947737 32681 ts_tablet_manager.cc:1428] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:44:41.948165 32394 heartbeater.cc:503] Master 127.21.112.126:44659 was elected leader, sending a full tablet report...
I20260928 23:44:41.951215 32680 raft_consensus.cc:348] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.951490 32680 raft_consensus.cc:374] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:44:41.951555 32680 raft_consensus.cc:729] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccec7bacd1be4f75aaa6bf2316605df7, State: Initialized, Role: FOLLOWER
I20260928 23:44:41.952034 32680 consensus_queue.cc:260] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.952778 32528 heartbeater.cc:503] Master 127.21.112.126:44659 was elected leader, sending a full tablet report...
I20260928 23:44:41.953339 32680 ts_tablet_manager.cc:1428] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:44:41.988204 32685 raft_consensus.cc:482] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:44:41.988459 32685 raft_consensus.cc:504] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.989828 32685 leader_election.cc:288] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccec7bacd1be4f75aaa6bf2316605df7 (127.21.112.66:41571), 4e71cc9cb9bf46cc90b89e5ba4e907bb (127.21.112.65:36363)
I20260928 23:44:41.994091 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.990227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:33621 (local address 127.21.112.65:36363)
0928 23:44:41.990368 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.990373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.990390 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:41.990478 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.990481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.990540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.990617 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:41.991082 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.991795 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.993107 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.993523 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.993684 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.993726 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.993923 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.993992 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:44:41.994135 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.990127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:36363 (local address 127.21.112.67:33621)
0928 23:44:41.990242 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.990259 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.990361 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.990686 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.990691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.990718 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:41.990905 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.990913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.991905 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.991908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.992986 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.992997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.993672 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.993676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.993791 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.993819 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.993902 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.993985 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"spinlock_wait_cycles":41728}
I20260928 23:44:41.994841 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.990470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:52531 (local address 127.21.112.66:41571)
0928 23:44:41.990675 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:41.990680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:41.990728 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:44:41.990896 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:41.990899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:41.990968 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:41.991054 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:44:41.991350 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.992030 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.993418 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.994139 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.994263 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.994323 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:41.994654 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 23:44:41.994727 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 23:44:41.994804 32348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4412a0aa512b40718ab92d0fb830225e" candidate_uuid: "db1d870dc0114f15993352ac1ac46a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" is_pre_election: true
I20260928 23:44:41.994865 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:41.990127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:41571 (local address 127.21.112.67:52531)
0928 23:44:41.990655 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:41.990668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:44:41.990788 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:44:41.991075 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:44:41.991078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:44:41.991095 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:44:41.991205 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.991217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.992161 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.992165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.993286 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:44:41.993295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:44:41.994388 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:44:41.994393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:44:41.994514 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:41.994534 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:44:41.994634 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 23:44:41.994707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":67,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260928 23:44:41.995147 32348 raft_consensus.cc:2454] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1d870dc0114f15993352ac1ac46a18 in term 0.
I20260928 23:44:41.995669 32551 leader_election.cc:302] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e71cc9cb9bf46cc90b89e5ba4e907bb, db1d870dc0114f15993352ac1ac46a18; no voters:
I20260928 23:44:41.995983 32685 raft_consensus.cc:2781] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:44:41.995972 32482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4412a0aa512b40718ab92d0fb830225e" candidate_uuid: "db1d870dc0114f15993352ac1ac46a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" is_pre_election: true
I20260928 23:44:41.996080 32685 raft_consensus.cc:482] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:44:41.996127 32685 raft_consensus.cc:3037] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:41.996271 32482 raft_consensus.cc:2454] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1d870dc0114f15993352ac1ac46a18 in term 0.
I20260928 23:44:41.997341 32685 raft_consensus.cc:504] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:41.997745 32685 leader_election.cc:288] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [CANDIDATE]: Term 1 election: Requested vote from peers ccec7bacd1be4f75aaa6bf2316605df7 (127.21.112.66:41571), 4e71cc9cb9bf46cc90b89e5ba4e907bb (127.21.112.65:36363)
I20260928 23:44:41.998083 32482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4412a0aa512b40718ab92d0fb830225e" candidate_uuid: "db1d870dc0114f15993352ac1ac46a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccec7bacd1be4f75aaa6bf2316605df7"
I20260928 23:44:41.998204 32482 raft_consensus.cc:3037] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:41.999185 32482 raft_consensus.cc:2454] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1d870dc0114f15993352ac1ac46a18 in term 1.
I20260928 23:44:41.999454 32348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4412a0aa512b40718ab92d0fb830225e" candidate_uuid: "db1d870dc0114f15993352ac1ac46a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb"
I20260928 23:44:41.999574 32552 leader_election.cc:302] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccec7bacd1be4f75aaa6bf2316605df7, db1d870dc0114f15993352ac1ac46a18; no voters:
I20260928 23:44:41.999593 32348 raft_consensus.cc:3037] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:44:41.999763 32685 raft_consensus.cc:2781] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:44:42.000012 32685 raft_consensus.cc:686] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 1 LEADER]: Becoming Leader. State: Replica: db1d870dc0114f15993352ac1ac46a18, State: Running, Role: LEADER
I20260928 23:44:42.000385 32685 consensus_queue.cc:237] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } }
I20260928 23:44:42.000943 32348 raft_consensus.cc:2454] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1d870dc0114f15993352ac1ac46a18 in term 1.
I20260928 23:44:42.003918 32197 catalog_manager.cc:5495] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1d870dc0114f15993352ac1ac46a18 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "db1d870dc0114f15993352ac1ac46a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db1d870dc0114f15993352ac1ac46a18" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 41711 } health_report { overall_health: HEALTHY } } }
I20260928 23:44:42.030138 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:42.025828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.21.112.67:41711)
0928 23:44:42.026007 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:44:42.026023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:44:42.026040 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:44:42.026226 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:44:42.026229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:44:42.026300 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:44:42.026509 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:44:42.026786 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:42.027352 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:42.028421 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:44:42.028674 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:44:42.028796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:44:42.029152 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:44:42.029699 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:44:42.029842 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:44:42.029910 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:44:42.029971 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:44:42.036897 32348 raft_consensus.cc:1261] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 1 FOLLOWER]: Refusing update from remote peer db1d870dc0114f15993352ac1ac46a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:42.036926 32482 raft_consensus.cc:1261] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 1 FOLLOWER]: Refusing update from remote peer db1d870dc0114f15993352ac1ac46a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:44:42.037722 32685 consensus_queue.cc:1035] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccec7bacd1be4f75aaa6bf2316605df7" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:42.037941 32689 consensus_queue.cc:1035] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e71cc9cb9bf46cc90b89e5ba4e907bb" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:44:42.046790 32694 mvcc.cc:205] Tried to move back new op lower bound from 7334457680014442496 to 7334457679876984832. Current Snapshot: MvccSnapshot[applied={T|T < 7334457680014442496}]
I20260928 23:44:42.056558 32203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:42.056727 32203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:44:42.059369 32203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c140be21d8e842e0844eb189165f4ec5: Sending DeleteTablet for 3 replicas of tablet 4412a0aa512b40718ab92d0fb830225e
I20260928 23:44:42.060250 32328 tablet_service.cc:1514] Processing DeleteTablet for tablet 4412a0aa512b40718ab92d0fb830225e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:42 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:49774
I20260928 23:44:42.060269 32462 tablet_service.cc:1514] Processing DeleteTablet for tablet 4412a0aa512b40718ab92d0fb830225e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:42 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:54886
I20260928 23:44:42.060570 32596 tablet_service.cc:1514] Processing DeleteTablet for tablet 4412a0aa512b40718ab92d0fb830225e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:44:42 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:44232
I20260928 23:44:42.060812 32702 tablet_replica.cc:330] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb: stopping tablet replica
I20260928 23:44:42.061012 32704 tablet_replica.cc:330] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: stopping tablet replica
I20260928 23:44:42.061156 32702 raft_consensus.cc:2229] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:44:42.061231 32704 raft_consensus.cc:2229] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:44:42.061492 32702 raft_consensus.cc:2258] T 4412a0aa512b40718ab92d0fb830225e P 4e71cc9cb9bf46cc90b89e5ba4e907bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:42.061677 32704 raft_consensus.cc:2258] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:42.062167 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 32265
I20260928 23:44:42.065578 32704 ts_tablet_manager.cc:1905] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:44:42.068100 32704 ts_tablet_manager.cc:1918] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:44:42.068218 32704 log.cc:1196] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/4412a0aa512b40718ab92d0fb830225e
I20260928 23:44:42.068514 32704 ts_tablet_manager.cc:1939] T 4412a0aa512b40718ab92d0fb830225e P db1d870dc0114f15993352ac1ac46a18: Deleting consensus metadata
I20260928 23:44:42.069788 32703 tablet_replica.cc:330] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7: stopping tablet replica
I20260928 23:44:42.069773 32188 catalog_manager.cc:4841] TS db1d870dc0114f15993352ac1ac46a18 (127.21.112.67:41711): tablet 4412a0aa512b40718ab92d0fb830225e (table test-table [id=52b68f38ffd74abfb41dc13ff028dc27]) successfully deleted
I20260928 23:44:42.070135 32703 raft_consensus.cc:2229] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 23:44:42.070374 32189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:36363 (error 108)
I20260928 23:44:42.070503 32703 raft_consensus.cc:2258] T 4412a0aa512b40718ab92d0fb830225e P ccec7bacd1be4f75aaa6bf2316605df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:44:42.070595 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 32399
I20260928 23:44:42.071939 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:42.071627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:36363 (local address 127.0.0.1:49786)
0928 23:44:42.071764 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:42.071845 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:36363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 23:44:42.072242 32189 catalog_manager.cc:4584] TS 4e71cc9cb9bf46cc90b89e5ba4e907bb (127.21.112.65:36363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4412a0aa512b40718ab92d0fb830225e: Network error: Client connection negotiation failed: client connection to 127.21.112.65:36363: connect: Connection refused (error 111)
I20260928 23:44:42.075941 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 32533
I20260928 23:44:42.076177 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:44:42.075870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:41571 (local address 127.0.0.1:54892)
0928 23:44:42.076020 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:44:42.076085 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.66:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 23:44:42.081061 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 32171
2026-09-28T23:44:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2439 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:44:42 dist-test-slave-c3s8 krb5kdc[32708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:44:42 dist-test-slave-c3s8 krb5kdc[32708](info): set up 2 sockets
Sep 28 23:44:42 dist-test-slave-c3s8 krb5kdc[32708](info): commencing operation
krb5kdc: starting...
W20260928 23:44:43.139091 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:44:43 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:44:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/rangeradmin_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/rangeradmin_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/rangerlookup_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/rangerlookup_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
I20260928 23:44:43.321158 21953 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzqclLI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/postgres -l logfile start
2026-09-28 23:44:46.164 UTC [32738] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:44:46.164 UTC [32738] LOG: listening on IPv4 address "127.21.112.84", port 44531
2026-09-28 23:44:46.169 UTC [32738] LOG: listening on Unix socket "/tmp/.s.PGSQL.44531"
2026-09-28 23:44:46.195 UTC [32741] LOG: database system was shut down at 2026-09-28 23:44:44 UTC
2026-09-28 23:44:46.199 UTC [32738] LOG: database system is ready to accept connections
I20260928 23:44:47.226715 21953 mini_postgres.cc:96] Postgres bound to 44531
2026-09-28 23:44:47.233 UTC [32749] FATAL: database "slave" does not exist
127.21.112.84:44531 - accepting connections
I20260928 23:44:47.234225 21953 mini_ranger.cc:161] Starting Ranger...
I20260928 23:44:47.250710 21953 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:44:47.927964 21953 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:44:47.928067 21953 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-admin
2026-09-28 23:44:48,299 [I] DB FLAVOR :POSTGRES
2026-09-28 23:44:48,300 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:44:48,300 [I] Checking connection..
2026-09-28 23:44:48,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:44:48,564 [I] Checking connection passed.
2026-09-28 23:44:48,564 [I] --------- Verifying version history table ---------
2026-09-28 23:44:48,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:44:48,802 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:44:48,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:44:49,048 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:44:49,048 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:44:49,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:44:49.297 UTC [345] WARNING: there is no transaction in progress
2026-09-28 23:44:49,304 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:44:49,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:44:49,543 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:44:49,543 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:44:49,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:44:49,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:44:50,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c3s8.c.gcp-upstream.internal','N') ;"
2026-09-28 23:44:50,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:44:50,536 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:44:50,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:44:50,776 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:44:50,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:44:51,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:44:51,292 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:44:51,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:44:52.347 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.368 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.387 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.405 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.425 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.446 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.512 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.520 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.536 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.548 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.560 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.575 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.587 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.595 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.604 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:52.612 UTC [453] WARNING: there is no transaction in progress
2026-09-28 23:44:53,403 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:44:53,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:44:53,641 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:44:53,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:44:53,887 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:44:53,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:44:54,153 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:44:54,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:44:54,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:44:54,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c3s8.c.gcp-upstream.internal';"
2026-09-28 23:44:54,885 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:44:54,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:44:55,139 [I] DB_PATCHES have already been applied
I20260928 23:44:55.145320 21953 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzqclLI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:44:55.145394 21953 mini_ranger.cc:193] Using host: 127.21.112.84
I20260928 23:44:55.149922 21953 mini_ranger.cc:227] Ranger admin URL: http://127.21.112.84:53055
Sep 28, 2026 11:44:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:44:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:44:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:44:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:44:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:44:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:44:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:44:56.282163 21953 mini_ranger.cc:160] Time spent starting Ranger: real 9.048s user 0.003s sys 0.008s
Sep 28, 2026 11:45:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:45:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 28, 2026 11:45:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:45:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:45:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:45:19.186053 21953 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/rangerkms_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/rangerkms_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.112.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.112.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:45:19 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:45:19.282095 21953 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:45:19.292505 21953 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:45:20.387585 21953 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:45:20.387696 21953 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms
I20260928 23:45:20.387814 21953 mini_ranger_kms.cc:227] Using postgres at 127.21.112.84:44531
2026-09-28 23:45:20,631 [I] DB FLAVOR :POSTGRES
2026-09-28 23:45:20,632 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:45:20,632 [I] Checking connection
2026-09-28 23:45:20,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:45:20,886 [I] connection success
2026-09-28 23:45:20,886 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:45:20,886 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:45:20,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:45:21,137 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:45:21,138 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:45:21,138 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:45:21,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:44531/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:45:21,414 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:45:21.936007 21953 mini_ranger_kms.cc:310] Created kms service
I20260928 23:45:22.151667 21953 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:45:22.205188 21953 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:45:22 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639119, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:45:22.650034 21953 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:45:22.650199 21953 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:45:22.650228 21953 mini_ranger_kms.cc:240] Using host: 127.21.112.84
I20260928 23:45:22.654069 21953 mini_ranger_kms.cc:276] Ranger KMS PID: 634
I20260928 23:45:22.654219 21953 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.112.84:34059
Sep 28, 2026 11:45:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:45:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:45:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:45:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:45:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:45:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:45:24.922006 21953 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.640s user 0.006s sys 0.171s
I20260928 23:45:24.922142 21953 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:45:24.922253 21953 mini_ranger_kms.cc:401] 127.21.112.84:34059/kms/v1/keys
Sep 28 23:45:25 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.112.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:45:25 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.112.84@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
Sep 28 23:45:25 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.112.84@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
Sep 28 23:45:28 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:45:28.062235 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37229
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:37229
--ranger_config_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:34059/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:45:28.179508 684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:45:28.179796 684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:45:28.179839 684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:45:28.183990 684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:45:28.184058 684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:45:28.184077 684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:45:28.184094 684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:45:28.184109 684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:45:28.189765 684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.112.84:34059/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:37229
--ranger_config_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:45:28.190814 684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:45:28.191846 684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:45:28.201323 689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:28.201409 690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:28.201323 692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:45:28.201483 684 server_base.cc:1034] running on GCE node
I20260928 23:45:28.202252 684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:45:28.203377 684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:45:28.204540 684 hybrid_clock.cc:648] HybridClock initialized: now 1790639128204513 us; error 39 us; skew 500 ppm
Sep 28 23:45:28 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639128, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:45:28.207526 684 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:45:28.208534 684 webserver.cc:460] Webserver started at http://127.21.112.126:45997/ using document root <none> and password file <none>
I20260928 23:45:28.209105 684 fs_manager.cc:359] Metadata directory not provided
I20260928 23:45:28.209160 684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:45:28.209409 684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:45:28 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639128, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:45:28.257215 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "0418937625e94de2a0f0065bc9b4282c"
format_stamp: "Formatted at 2026-09-28 23:45:28 on dist-test-slave-c3s8"
server_key: "1b4a99b7701ac41b60f051e0c17e752e"
server_key_iv: "4f2e36d55936cf775e2f7679d4334847"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:28.257912 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "0418937625e94de2a0f0065bc9b4282c"
format_stamp: "Formatted at 2026-09-28 23:45:28 on dist-test-slave-c3s8"
server_key: "1b4a99b7701ac41b60f051e0c17e752e"
server_key_iv: "4f2e36d55936cf775e2f7679d4334847"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:28.262172 684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:45:28.280525 699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:28.282387 684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:45:28.282545 684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "0418937625e94de2a0f0065bc9b4282c"
format_stamp: "Formatted at 2026-09-28 23:45:28 on dist-test-slave-c3s8"
server_key: "1b4a99b7701ac41b60f051e0c17e752e"
server_key_iv: "4f2e36d55936cf775e2f7679d4334847"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:28.282696 684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:45:28.324019 684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:45:28.327883 684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:45:28.328137 684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:45:28.336335 684 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:37229
I20260928 23:45:28.336357 751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:37229 every 8 connection(s)
I20260928 23:45:28.337320 684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:45:28.340261 752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:45:28.340945 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 684
I20260928 23:45:28.341099 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:45:28.341395 21953 external_mini_cluster.cc:1428] Setting key 3160b39d5a30ee314ada7bcaeb545f04
I20260928 23:45:28.346973 752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c: Bootstrap starting.
I20260928 23:45:28.349802 752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c: Neither blocks nor log segments found. Creating new log.
I20260928 23:45:28.350730 752 log.cc:824] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:45:28 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:45:28.353364 752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c: No bootstrap required, opened a new log
I20260928 23:45:28.357602 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:28.342845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.21.112.126:37229)
0928 23:45:28.343279 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:28.343296 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:28.343332 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:45:28.343975 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:28.343992 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:28.344331 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:28.344623 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:45:28.345726 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:28.346710 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:28.347899 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:28.348391 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:28.348525 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:28.351713 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:45:28.351720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:45:28.351738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:45:28.351766 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:45:28.354350 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:28.354807 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:28.354821 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:28.354828 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:28.354961 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:28.355246 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:28.355249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:28.355251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:28.355765 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:45:28.355980 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:28.356292 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 23:45:28.356587 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":145,"threads_started":1}
I20260928 23:45:28.357599 752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418937625e94de2a0f0065bc9b4282c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37229 } }
I20260928 23:45:28.357949 752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:45:28.358014 752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0418937625e94de2a0f0065bc9b4282c, State: Initialized, Role: FOLLOWER
I20260928 23:45:28.358444 752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418937625e94de2a0f0065bc9b4282c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37229 } }
I20260928 23:45:28.358596 752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:45:28.358666 752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:45:28.358763 752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:45:28.359992 752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418937625e94de2a0f0065bc9b4282c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37229 } }
I20260928 23:45:28.360389 752 leader_election.cc:302] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0418937625e94de2a0f0065bc9b4282c; no voters:
I20260928 23:45:28.360740 752 leader_election.cc:288] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:45:28.360920 757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:45:28.361153 757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [term 1 LEADER]: Becoming Leader. State: Replica: 0418937625e94de2a0f0065bc9b4282c, State: Running, Role: LEADER
I20260928 23:45:28.361593 757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418937625e94de2a0f0065bc9b4282c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37229 } }
I20260928 23:45:28.362094 752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:45:28.363384 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0418937625e94de2a0f0065bc9b4282c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418937625e94de2a0f0065bc9b4282c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37229 } } }
I20260928 23:45:28.363445 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0418937625e94de2a0f0065bc9b4282c. Latest consensus state: current_term: 1 leader_uuid: "0418937625e94de2a0f0065bc9b4282c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0418937625e94de2a0f0065bc9b4282c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37229 } } }
I20260928 23:45:28.363536 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [sys.catalog]: This master's current role is: LEADER
I20260928 23:45:28.363536 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c [sys.catalog]: This master's current role is: LEADER
I20260928 23:45:28.364120 769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:45:28.364387 752 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:45:28.369563 769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:45:28.376772 769 catalog_manager.cc:1347] Generated new cluster ID: 747f5569f8c548fa957106f33fde3e9b
I20260928 23:45:28.376842 769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:45:28.395782 769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:45:28.397217 769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:45:28.403669 769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c: Generated new TSK 0
I20260928 23:45:28.404603 769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:45:29.151081 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:34059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37229
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:45:29.299723 797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:45:29.300182 797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:45:29.300251 797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:45:29.306697 797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:45:29.306802 797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:45:29.306890 797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:45:29.315311 797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.112.84:34059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:45:29.316537 797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:45:29.317783 797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:45:29.328233 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:29.328233 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:29.328343 804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:45:29.329876 797 server_base.cc:1034] running on GCE node
I20260928 23:45:29.330535 797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:45:29.331384 797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:45:29.334661 797 hybrid_clock.cc:648] HybridClock initialized: now 1790639129334631 us; error 37 us; skew 500 ppm
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:45:29.338809 797 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:45:29.340032 797 webserver.cc:460] Webserver started at http://127.21.112.65:38617/ using document root <none> and password file <none>
I20260928 23:45:29.340837 797 fs_manager.cc:359] Metadata directory not provided
I20260928 23:45:29.340987 797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:45:29.341277 797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:45:29.366815 797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "76b784b7978a4835ae8c3e72b0e1e685"
format_stamp: "Formatted at 2026-09-28 23:45:29 on dist-test-slave-c3s8"
server_key: "f3151bd11e3bc68232f4d959b9365b4c"
server_key_iv: "0c1007a10d5084f1203371185cb1a157"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:29.367492 797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "76b784b7978a4835ae8c3e72b0e1e685"
format_stamp: "Formatted at 2026-09-28 23:45:29 on dist-test-slave-c3s8"
server_key: "f3151bd11e3bc68232f4d959b9365b4c"
server_key_iv: "0c1007a10d5084f1203371185cb1a157"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:29.374209 797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 23:45:29.392373 816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:29.394136 797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:45:29.394275 797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "76b784b7978a4835ae8c3e72b0e1e685"
format_stamp: "Formatted at 2026-09-28 23:45:29 on dist-test-slave-c3s8"
server_key: "f3151bd11e3bc68232f4d959b9365b4c"
server_key_iv: "0c1007a10d5084f1203371185cb1a157"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:29.394428 797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:45:29.417285 797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:45:29.421103 797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:45:29.421384 797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:45:29.422145 797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:45:29.423244 797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:45:29.423358 797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:29.423483 797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:45:29.423563 797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:29.434623 797 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:38787
I20260928 23:45:29.434749 929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:38787 every 8 connection(s)
I20260928 23:45:29.435930 797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:45:29.444483 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 797
I20260928 23:45:29.444630 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:45:29.444940 21953 external_mini_cluster.cc:1428] Setting key d93f31fb3411eca818def373931c7166
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:45:29.459527 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:29.438942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:40925 (local address 127.21.112.126:37229)
0928 23:45:29.439267 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:29.439272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:29.439747 (+ 475us) server_negotiation.cc:407] Connection header received
0928 23:45:29.441115 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:29.441119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:29.441184 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:29.441290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:45:29.442867 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.443738 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.449581 (+ 5843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.449954 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.450089 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:29.454158 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:45:29.454162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:45:29.454165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:45:29.454197 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:45:29.456583 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:29.457272 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:29.457276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:29.457278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:29.457387 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:29.457877 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:29.457881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:29.457883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:29.458180 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:45:29.458286 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:29.459106 (+ 820us) server_negotiation.cc:299] Negotiation successful
0928 23:45:29.459336 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:45:29.460636 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:29.438709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37229 (local address 127.21.112.65:40925)
0928 23:45:29.439569 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:29.439631 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:29.440743 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:29.441592 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:29.441603 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:29.441968 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:45:29.442677 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:29.442697 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.443898 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.443901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:29.449370 (+ 5469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:29.449387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.450921 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.450927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:29.451169 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:29.451988 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:45:29.452013 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:45:29.453909 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:45:29.456736 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:45:29.456743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:45:29.456764 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:45:29.457145 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:45:29.457510 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:45:29.457513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:45:29.457516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:45:29.457742 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:45:29.458345 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:45:29.458353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:45:29.458737 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 23:45:29.458984 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 23:45:29.459459 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":144,"threads_started":1}
I20260928 23:45:29.463552 930 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37229
I20260928 23:45:29.463828 930 heartbeater.cc:464] Registering TS with master...
I20260928 23:45:29.464460 930 heartbeater.cc:511] Master 127.21.112.126:37229 requested a full tablet report, sending...
I20260928 23:45:29.466375 716 ts_manager.cc:194] Registered new tserver with Master: 76b784b7978a4835ae8c3e72b0e1e685 (127.21.112.65:38787)
I20260928 23:45:29.467907 716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:40925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:45:29.484952 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:34059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37229
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:45:29.633669 936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:45:29.633975 936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:45:29.634096 936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:45:29.638513 936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:45:29.638602 936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:45:29.638706 936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:45:29.644287 936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.112.84:34059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:45:29.645421 936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:45:29.646479 936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:45:29.657621 943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:29.658147 942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:45:29.659245 936 server_base.cc:1034] running on GCE node
W20260928 23:45:29.659508 945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:45:29.660049 936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:45:29.660876 936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:45:29.662190 936 hybrid_clock.cc:648] HybridClock initialized: now 1790639129662114 us; error 95 us; skew 500 ppm
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:45:29.665937 936 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:45:29.667048 936 webserver.cc:460] Webserver started at http://127.21.112.66:39445/ using document root <none> and password file <none>
I20260928 23:45:29.667744 936 fs_manager.cc:359] Metadata directory not provided
I20260928 23:45:29.667819 936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:45:29.668061 936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:45:29.681931 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "d3ecba4ee94b4564a3413acbdad57842"
format_stamp: "Formatted at 2026-09-28 23:45:29 on dist-test-slave-c3s8"
server_key: "5a46f460e9f28585a330e4719bf14031"
server_key_iv: "a552ff43fa084fcb7a713b6af3768092"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:29.682763 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "d3ecba4ee94b4564a3413acbdad57842"
format_stamp: "Formatted at 2026-09-28 23:45:29 on dist-test-slave-c3s8"
server_key: "5a46f460e9f28585a330e4719bf14031"
server_key_iv: "a552ff43fa084fcb7a713b6af3768092"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:29.687811 936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 23:45:29.705214 952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:29.707216 936 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260928 23:45:29.707379 936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "d3ecba4ee94b4564a3413acbdad57842"
format_stamp: "Formatted at 2026-09-28 23:45:29 on dist-test-slave-c3s8"
server_key: "5a46f460e9f28585a330e4719bf14031"
server_key_iv: "a552ff43fa084fcb7a713b6af3768092"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:29.707538 936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:45:29.723410 936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:45:29.735899 936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:45:29.736184 936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:45:29.737013 936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:45:29.738082 936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:45:29.738195 936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:29.738306 936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:45:29.738377 936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:29.755465 936 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:44433
I20260928 23:45:29.755512 1065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:44433 every 8 connection(s)
I20260928 23:45:29.756763 936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:45:29.760442 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 936
I20260928 23:45:29.760591 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:45:29.760912 21953 external_mini_cluster.cc:1428] Setting key 706cde4ac3d8afaf891ace5bb1db6a1b
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:45:29.776199 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:29.759532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:53895 (local address 127.21.112.126:37229)
0928 23:45:29.759674 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:29.759679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:29.760925 (+ 1246us) server_negotiation.cc:407] Connection header received
0928 23:45:29.763652 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:29.763656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:29.763716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:29.763774 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:45:29.765617 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.766457 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.768215 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.768586 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.768667 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:29.771842 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:45:29.771846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:45:29.771849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:45:29.771877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:45:29.773714 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:29.774376 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:29.774379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:29.774381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:29.774472 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:29.774872 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:29.774875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:29.774877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:29.775112 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:45:29.775182 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:29.775882 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 23:45:29.776023 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:45:29.777245 1068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:29.759430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37229 (local address 127.21.112.66:53895)
0928 23:45:29.760740 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:29.760799 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:29.763319 (+ 2520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:29.764238 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:29.764248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:29.764637 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:45:29.765403 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:29.765425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.766627 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.766631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:29.768067 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:29.768079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:29.768894 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:29.768898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:29.769104 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:29.769838 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:45:29.769859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:45:29.771638 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:45:29.773888 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:45:29.773894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:45:29.773913 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:45:29.774258 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:45:29.774581 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:45:29.774584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:45:29.774586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:45:29.774759 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:45:29.775265 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:45:29.775272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:45:29.775590 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 23:45:29.775805 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 23:45:29.776153 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:45:29.778829 1066 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37229
I20260928 23:45:29.779089 1066 heartbeater.cc:464] Registering TS with master...
I20260928 23:45:29.779661 1066 heartbeater.cc:511] Master 127.21.112.126:37229 requested a full tablet report, sending...
I20260928 23:45:29.780825 716 ts_manager.cc:194] Registered new tserver with Master: d3ecba4ee94b4564a3413acbdad57842 (127.21.112.66:44433)
I20260928 23:45:29.781621 716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:53895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:45:29.801994 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:34059/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37229
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:45:29 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
W20260928 23:45:29.972695 1071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:45:29.973632 1071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:45:29.973798 1071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:45:29.980254 1071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:45:29.980484 1071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:45:29.980654 1071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:45:29.989638 1071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.112.84:34059/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:45:29.991432 1071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:45:29.992918 1071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:45:30.007431 1077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:30.008034 1079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:45:30.008801 1076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:45:30.009618 1071 server_base.cc:1034] running on GCE node
I20260928 23:45:30.010305 1071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:45:30.011232 1071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:45:30.012538 1071 hybrid_clock.cc:648] HybridClock initialized: now 1790639130012466 us; error 94 us; skew 500 ppm
Sep 28 23:45:30 dist-test-slave-c3s8 krb5kdc[32708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639130, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:45:30.016553 1071 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:45:30.017843 1071 webserver.cc:460] Webserver started at http://127.21.112.67:44649/ using document root <none> and password file <none>
I20260928 23:45:30.018685 1071 fs_manager.cc:359] Metadata directory not provided
I20260928 23:45:30.018829 1071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:45:30.019136 1071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:45:30 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639130, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:45:30.052605 1071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "3acda52d60304bcdb58a747000c692a8"
format_stamp: "Formatted at 2026-09-28 23:45:30 on dist-test-slave-c3s8"
server_key: "49f873b209a9f92a28e0ca565e92b60c"
server_key_iv: "ec3d2bb663d1a03bdc53858aa0380174"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:30.053606 1071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "3acda52d60304bcdb58a747000c692a8"
format_stamp: "Formatted at 2026-09-28 23:45:30 on dist-test-slave-c3s8"
server_key: "49f873b209a9f92a28e0ca565e92b60c"
server_key_iv: "ec3d2bb663d1a03bdc53858aa0380174"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:30.063944 1071 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.000s sys 0.005s
I20260928 23:45:30.080583 1086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:30.082414 1071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:45:30.082571 1071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "3acda52d60304bcdb58a747000c692a8"
format_stamp: "Formatted at 2026-09-28 23:45:30 on dist-test-slave-c3s8"
server_key: "49f873b209a9f92a28e0ca565e92b60c"
server_key_iv: "ec3d2bb663d1a03bdc53858aa0380174"
server_key_version: "kuduclusterkey@0"
I20260928 23:45:30.082715 1071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:45:30.112155 1071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:45:30.116182 1071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:45:30.116465 1071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:45:30.117218 1071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:45:30.118355 1071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:45:30.118456 1071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:30.118551 1071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:45:30.118613 1071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:45:30.129189 1071 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:43867
I20260928 23:45:30.131131 1071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:45:30.133793 1199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:43867 every 8 connection(s)
I20260928 23:45:30.134608 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 1071
I20260928 23:45:30.134754 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:45:30.135099 21953 external_mini_cluster.cc:1428] Setting key 63d259982383d30002cae07c74b89c26
Sep 28 23:45:30 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639130, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:45:30.151923 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.137882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:36917 (local address 127.21.112.126:37229)
0928 23:45:30.138018 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.138022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.138357 (+ 335us) server_negotiation.cc:407] Connection header received
0928 23:45:30.139559 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.139562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.139612 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.139661 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:45:30.141297 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.142151 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.143883 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.144229 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.144301 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.147491 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:45:30.147495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:45:30.147498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:45:30.147527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:45:30.149361 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:30.150045 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:30.150048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:30.150050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:30.150130 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:30.150523 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:30.150526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:30.150528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:30.150741 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:45:30.150818 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.151596 (+ 778us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.151748 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:45:30.153853 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.136669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37229 (local address 127.21.112.67:36917)
0928 23:45:30.138199 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.138254 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.139309 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.140094 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:30.140103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:30.140460 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:45:30.141133 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.141152 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.142323 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.142326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.143727 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.143737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.144504 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.144508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.144710 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.145440 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:45:30.145461 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:45:30.147279 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:45:30.149506 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:45:30.149512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:45:30.149566 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:45:30.149928 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:45:30.150231 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:45:30.150234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:45:30.150236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:45:30.150418 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:45:30.150894 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:45:30.150900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:45:30.151298 (+ 398us) client_negotiation.cc:769] Sending connection context
0928 23:45:30.152343 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0928 23:45:30.152713 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"thread_start_us":176,"threads_started":1}
I20260928 23:45:30.155371 1200 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37229
I20260928 23:45:30.155612 1200 heartbeater.cc:464] Registering TS with master...
I20260928 23:45:30.156176 1200 heartbeater.cc:511] Master 127.21.112.126:37229 requested a full tablet report, sending...
I20260928 23:45:30.157297 716 ts_manager.cc:194] Registered new tserver with Master: 3acda52d60304bcdb58a747000c692a8 (127.21.112.67:43867)
I20260928 23:45:30.158226 716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:36917
I20260928 23:45:30.161165 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:45:30.173633 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.163994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.21.112.126:37229)
0928 23:45:30.164109 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.164114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.164343 (+ 229us) server_negotiation.cc:407] Connection header received
0928 23:45:30.164522 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.164525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.164569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.164616 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:45:30.165614 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.166425 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.167642 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.167994 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.168064 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.169918 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:45:30.169921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:45:30.169924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:45:30.169951 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:45:30.171730 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:30.172247 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:30.172251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:30.172253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:30.172335 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:45:30.172624 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:45:30.172627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:45:30.172629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:45:30.172833 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:45:30.172955 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.173312 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.173433 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:45:30.177778 716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:45:30.182262 716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:45:30.218005 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.203481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:44433 (local address 127.0.0.1:37696)
0928 23:45:30.208524 (+ 5043us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.208536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.208601 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.213604 (+ 5003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:30.213608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:30.213625 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:45:30.213735 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.213741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.215291 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.215293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.216311 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.216318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.217611 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.217615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.217705 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.217728 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:45:30.217796 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:45:30.217863 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4986,"thread_start_us":85,"threads_started":1}
I20260928 23:45:30.219125 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.199477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:43867 (local address 127.0.0.1:40802)
0928 23:45:30.204870 (+ 5393us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.204935 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.205067 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.208214 (+ 3147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:30.208217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:30.208233 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:45:30.208329 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.208334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.216241 (+ 7907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.216244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.217376 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.217385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.218715 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.218719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.218809 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.218830 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:45:30.218912 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:45:30.218977 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5184,"thread_start_us":119,"threads_started":1}
I20260928 23:45:30.219561 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.202369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:38787 (local address 127.0.0.1:45494)
0928 23:45:30.205482 (+ 3113us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.205495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.205623 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.206368 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:30.206375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:30.206397 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:45:30.206529 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.206535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.212086 (+ 5551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.212090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.213604 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.213612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.214882 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.214886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.214991 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.215026 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:45:30.215128 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:45:30.215265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3058,"thread_start_us":85,"threads_started":1}
I20260928 23:45:30.220296 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.202757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.21.112.65:38787)
0928 23:45:30.205356 (+ 2599us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.205364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.205716 (+ 352us) server_negotiation.cc:407] Connection header received
0928 23:45:30.205801 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.205806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.206061 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.206204 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:45:30.208701 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.211958 (+ 3257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.213727 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.214560 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.214663 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.214807 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.220044 (+ 5237us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.220163 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2487,"thread_start_us":108,"threads_started":1}
I20260928 23:45:30.220762 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.200425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.21.112.67:43867)
0928 23:45:30.206808 (+ 6383us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.206819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.206854 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:45:30.206922 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.206927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.207928 (+ 1001us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.208074 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:45:30.209628 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.216126 (+ 6498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.217523 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.218409 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.218510 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.218655 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.220528 (+ 1873us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.220633 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6272,"thread_start_us":148,"threads_started":1}
I20260928 23:45:30.221887 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.203629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.21.112.66:44433)
0928 23:45:30.213121 (+ 9492us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.213136 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.213160 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:45:30.213232 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.213237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.213430 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.213899 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:45:30.214110 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.215141 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.216462 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.217259 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.217374 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.217557 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.221622 (+ 4065us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.221736 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9385,"thread_start_us":115,"threads_started":1}
I20260928 23:45:30.225034 863 tablet_service.cc:1467] Processing CreateTablet for tablet dc156bcf78b54ddcb7fa7f9e16b55619 (DEFAULT_TABLE table=test-table [id=a4704027da124e43a898515148617c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:45:30.225040 989 tablet_service.cc:1467] Processing CreateTablet for tablet dc156bcf78b54ddcb7fa7f9e16b55619 (DEFAULT_TABLE table=test-table [id=a4704027da124e43a898515148617c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:45:30.226240 863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc156bcf78b54ddcb7fa7f9e16b55619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:45:30.226243 989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc156bcf78b54ddcb7fa7f9e16b55619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:45:30.237828 1134 tablet_service.cc:1467] Processing CreateTablet for tablet dc156bcf78b54ddcb7fa7f9e16b55619 (DEFAULT_TABLE table=test-table [id=a4704027da124e43a898515148617c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:45:30.238993 1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc156bcf78b54ddcb7fa7f9e16b55619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:45:30.242249 1219 tablet_bootstrap.cc:492] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Bootstrap starting.
I20260928 23:45:30.248888 1221 tablet_bootstrap.cc:492] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Bootstrap starting.
I20260928 23:45:30.249926 1220 tablet_bootstrap.cc:492] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685: Bootstrap starting.
I20260928 23:45:30.251905 1219 tablet_bootstrap.cc:654] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Neither blocks nor log segments found. Creating new log.
I20260928 23:45:30.252861 1219 log.cc:824] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Log is configured to *not* fsync() on all Append() calls
I20260928 23:45:30.255321 1220 tablet_bootstrap.cc:654] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685: Neither blocks nor log segments found. Creating new log.
I20260928 23:45:30.255622 1221 tablet_bootstrap.cc:654] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Neither blocks nor log segments found. Creating new log.
I20260928 23:45:30.256524 1221 log.cc:824] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:45:30.257369 1220 log.cc:824] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685: Log is configured to *not* fsync() on all Append() calls
I20260928 23:45:30.260418 1221 tablet_bootstrap.cc:492] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: No bootstrap required, opened a new log
I20260928 23:45:30.260690 1221 ts_tablet_manager.cc:1397] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 23:45:30.263507 1219 tablet_bootstrap.cc:492] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: No bootstrap required, opened a new log
I20260928 23:45:30.263844 1219 ts_tablet_manager.cc:1397] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260928 23:45:30.266698 1221 raft_consensus.cc:348] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.267975 1219 raft_consensus.cc:348] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.268326 1219 raft_consensus.cc:374] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:45:30.268422 1219 raft_consensus.cc:729] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ecba4ee94b4564a3413acbdad57842, State: Initialized, Role: FOLLOWER
I20260928 23:45:30.268803 1220 tablet_bootstrap.cc:492] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685: No bootstrap required, opened a new log
I20260928 23:45:30.269101 1220 ts_tablet_manager.cc:1397] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260928 23:45:30.269040 1219 consensus_queue.cc:260] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.270210 1219 ts_tablet_manager.cc:1428] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:45:30.272193 1066 heartbeater.cc:503] Master 127.21.112.126:37229 was elected leader, sending a full tablet report...
I20260928 23:45:30.273788 1220 raft_consensus.cc:348] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.274232 1221 raft_consensus.cc:374] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:45:30.274346 1221 raft_consensus.cc:729] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acda52d60304bcdb58a747000c692a8, State: Initialized, Role: FOLLOWER
I20260928 23:45:30.274700 1220 raft_consensus.cc:374] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:45:30.274822 1220 raft_consensus.cc:729] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b784b7978a4835ae8c3e72b0e1e685, State: Initialized, Role: FOLLOWER
I20260928 23:45:30.275210 1221 consensus_queue.cc:260] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.275846 1220 consensus_queue.cc:260] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.277006 1221 ts_tablet_manager.cc:1428] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260928 23:45:30.277959 1220 ts_tablet_manager.cc:1428] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260928 23:45:30.278398 930 heartbeater.cc:503] Master 127.21.112.126:37229 was elected leader, sending a full tablet report...
I20260928 23:45:30.278586 1200 heartbeater.cc:503] Master 127.21.112.126:37229 was elected leader, sending a full tablet report...
Sep 28 23:45:30 dist-test-slave-c3s8 krb5kdc[32708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639129, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:45:30.359381 1226 raft_consensus.cc:482] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:45:30.359648 1226 raft_consensus.cc:504] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.360881 1226 leader_election.cc:288] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b784b7978a4835ae8c3e72b0e1e685 (127.21.112.65:38787), d3ecba4ee94b4564a3413acbdad57842 (127.21.112.66:44433)
I20260928 23:45:30.366503 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.361411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:44433 (local address 127.21.112.67:41607)
0928 23:45:30.361871 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.361884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.361980 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.362603 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:30.362606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:30.362626 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:45:30.362765 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.362773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.363801 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.363805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.365050 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.365061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.366052 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.366055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.366156 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.366179 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:45:30.366248 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:45:30.366325 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"spinlock_wait_cycles":19712,"thread_start_us":179,"threads_started":1}
I20260928 23:45:30.366730 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.362236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:41933 (local address 127.21.112.65:38787)
0928 23:45:30.362362 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.362365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.362378 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:45:30.362425 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.362427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.362464 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.362566 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:45:30.363390 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.364108 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.365460 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.366056 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.366144 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.366189 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.366516 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.366601 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:45:30.366870 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.362213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:41607 (local address 127.21.112.66:44433)
0928 23:45:30.362362 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.362367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.362386 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:45:30.362443 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.362447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.362496 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.362585 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:45:30.362933 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.363677 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.365202 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.365818 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.365938 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.365993 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.366669 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.366743 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:45:30.367930 1009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc156bcf78b54ddcb7fa7f9e16b55619" candidate_uuid: "3acda52d60304bcdb58a747000c692a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ecba4ee94b4564a3413acbdad57842" is_pre_election: true
I20260928 23:45:30.368155 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.361054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:38787 (local address 127.21.112.67:41933)
0928 23:45:30.361804 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.361821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.361941 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.363094 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:45:30.363098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:45:30.363120 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:45:30.363230 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.363235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.364245 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.364248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.365334 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:45:30.365343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.366284 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.366287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:45:30.366381 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.366400 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:45:30.367829 (+ 1429us) client_negotiation.cc:240] Negotiation successful
0928 23:45:30.367908 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662}
I20260928 23:45:30.368299 1009 raft_consensus.cc:2454] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3acda52d60304bcdb58a747000c692a8 in term 0.
I20260928 23:45:30.369041 884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc156bcf78b54ddcb7fa7f9e16b55619" candidate_uuid: "3acda52d60304bcdb58a747000c692a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b784b7978a4835ae8c3e72b0e1e685" is_pre_election: true
I20260928 23:45:30.369347 884 raft_consensus.cc:2454] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3acda52d60304bcdb58a747000c692a8 in term 0.
I20260928 23:45:30.369961 1087 leader_election.cc:302] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acda52d60304bcdb58a747000c692a8, d3ecba4ee94b4564a3413acbdad57842; no voters:
I20260928 23:45:30.370299 1226 raft_consensus.cc:2781] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:45:30.370391 1226 raft_consensus.cc:482] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:45:30.370435 1226 raft_consensus.cc:3037] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:45:30.371663 1226 raft_consensus.cc:504] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.372010 1226 leader_election.cc:288] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [CANDIDATE]: Term 1 election: Requested vote from peers 76b784b7978a4835ae8c3e72b0e1e685 (127.21.112.65:38787), d3ecba4ee94b4564a3413acbdad57842 (127.21.112.66:44433)
I20260928 23:45:30.372553 884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc156bcf78b54ddcb7fa7f9e16b55619" candidate_uuid: "3acda52d60304bcdb58a747000c692a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b784b7978a4835ae8c3e72b0e1e685"
I20260928 23:45:30.372556 1009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc156bcf78b54ddcb7fa7f9e16b55619" candidate_uuid: "3acda52d60304bcdb58a747000c692a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ecba4ee94b4564a3413acbdad57842"
I20260928 23:45:30.372680 884 raft_consensus.cc:3037] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:45:30.372681 1009 raft_consensus.cc:3037] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:45:30.374053 1009 raft_consensus.cc:2454] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3acda52d60304bcdb58a747000c692a8 in term 1.
I20260928 23:45:30.374053 884 raft_consensus.cc:2454] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3acda52d60304bcdb58a747000c692a8 in term 1.
I20260928 23:45:30.374506 1087 leader_election.cc:302] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acda52d60304bcdb58a747000c692a8, 76b784b7978a4835ae8c3e72b0e1e685; no voters:
I20260928 23:45:30.374689 1226 raft_consensus.cc:2781] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:45:30.374936 1226 raft_consensus.cc:686] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 1 LEADER]: Becoming Leader. State: Replica: 3acda52d60304bcdb58a747000c692a8, State: Running, Role: LEADER
I20260928 23:45:30.375344 1226 consensus_queue.cc:237] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } }
I20260928 23:45:30.379367 714 catalog_manager.cc:5495] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3acda52d60304bcdb58a747000c692a8 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "3acda52d60304bcdb58a747000c692a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acda52d60304bcdb58a747000c692a8" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 43867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 } health_report { overall_health: UNKNOWN } } }
W20260928 23:45:30.385390 1201 tablet.cc:2369] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:45:30.402876 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.398020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.21.112.67:43867)
0928 23:45:30.398138 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:45:30.398143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:45:30.398205 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:45:30.398420 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:45:30.398426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:45:30.398503 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:45:30.398614 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:45:30.398887 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.399558 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.400787 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:45:30.401137 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:45:30.401240 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:45:30.402010 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:45:30.402451 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:45:30.402590 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:45:30.402626 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:45:30.402688 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:45:30.410585 1009 raft_consensus.cc:1261] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 1 FOLLOWER]: Refusing update from remote peer 3acda52d60304bcdb58a747000c692a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:45:30.411453 884 raft_consensus.cc:1261] T dc156bcf78b54ddcb7fa7f9e16b55619 P 76b784b7978a4835ae8c3e72b0e1e685 [term 1 FOLLOWER]: Refusing update from remote peer 3acda52d60304bcdb58a747000c692a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:45:30.411885 1226 consensus_queue.cc:1035] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ecba4ee94b4564a3413acbdad57842" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:45:30.412060 1230 consensus_queue.cc:1035] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b784b7978a4835ae8c3e72b0e1e685" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:45:30.426478 1234 mvcc.cc:205] Tried to move back new op lower bound from 7334457878149992448 to 7334457878022033408. Current Snapshot: MvccSnapshot[applied={T|T < 7334457878149992448}]
I20260928 23:45:30.428660 1236 mvcc.cc:205] Tried to move back new op lower bound from 7334457878149992448 to 7334457878022033408. Current Snapshot: MvccSnapshot[applied={T|T < 7334457878149992448}]
I20260928 23:45:30.432206 1235 mvcc.cc:205] Tried to move back new op lower bound from 7334457878149992448 to 7334457878022033408. Current Snapshot: MvccSnapshot[applied={T|T < 7334457878149992448}]
I20260928 23:45:30.444919 714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:45:30.445119 714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:45:30.448233 714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0418937625e94de2a0f0065bc9b4282c: Sending DeleteTablet for 3 replicas of tablet dc156bcf78b54ddcb7fa7f9e16b55619
I20260928 23:45:30.449995 1134 tablet_service.cc:1514] Processing DeleteTablet for tablet dc156bcf78b54ddcb7fa7f9e16b55619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:45:30 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:40802
I20260928 23:45:30.450546 863 tablet_service.cc:1514] Processing DeleteTablet for tablet dc156bcf78b54ddcb7fa7f9e16b55619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:45:30 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:45494
I20260928 23:45:30.451004 989 tablet_service.cc:1514] Processing DeleteTablet for tablet dc156bcf78b54ddcb7fa7f9e16b55619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:45:30 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37696
I20260928 23:45:30.451947 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 797
I20260928 23:45:30.456511 1245 tablet_replica.cc:330] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: stopping tablet replica
I20260928 23:45:30.460425 1245 raft_consensus.cc:2229] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:45:30.461205 1243 tablet_replica.cc:330] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: stopping tablet replica
I20260928 23:45:30.462946 1245 raft_consensus.cc:2258] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:45:30.462311 1243 raft_consensus.cc:2229] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:45:30.466796 1243 raft_consensus.cc:2258] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:45:30.471493 1243 ts_tablet_manager.cc:1905] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:45:30.471880 1245 ts_tablet_manager.cc:1905] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:45:30.477504 1245 ts_tablet_manager.cc:1918] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:45:30.480372 1243 ts_tablet_manager.cc:1918] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 23:45:30.480944 700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:38787 (error 108)
I20260928 23:45:30.481904 1245 log.cc:1196] T dc156bcf78b54ddcb7fa7f9e16b55619 P d3ecba4ee94b4564a3413acbdad57842: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/dc156bcf78b54ddcb7fa7f9e16b55619
I20260928 23:45:30.482234 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 936
I20260928 23:45:30.489692 1243 log.cc:1196] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/dc156bcf78b54ddcb7fa7f9e16b55619
I20260928 23:45:30.490212 1243 ts_tablet_manager.cc:1939] T dc156bcf78b54ddcb7fa7f9e16b55619 P 3acda52d60304bcdb58a747000c692a8: Deleting consensus metadata
I20260928 23:45:30.491672 700 catalog_manager.cc:4841] TS 3acda52d60304bcdb58a747000c692a8 (127.21.112.67:43867): tablet dc156bcf78b54ddcb7fa7f9e16b55619 (table test-table [id=a4704027da124e43a898515148617c46]) successfully deleted
I20260928 23:45:30.492782 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:45:30.492456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:38787 (local address 127.0.0.1:45510)
0928 23:45:30.492629 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.492698 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:38787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 23:45:30.493054 700 catalog_manager.cc:4584] TS 76b784b7978a4835ae8c3e72b0e1e685 (127.21.112.65:38787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc156bcf78b54ddcb7fa7f9e16b55619: Network error: Client connection negotiation failed: client connection to 127.21.112.65:38787: connect: Connection refused (error 111)
W20260928 23:45:30.496706 1217 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:45:30.496175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:44433 (local address 127.0.0.1:37698)
0928 23:45:30.496324 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:45:30.496338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:45:30.496421 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:45:30.496617 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.66:44433: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:45:30.497359 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 1071
I20260928 23:45:30.504720 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 684
I20260928 23:45:30.510339 21953 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:45:30.855990 21953 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:45:30.856115 21953 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:45:31.201647 21953 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:45:31.202 UTC [32738] LOG: received smart shutdown request
2026-09-28 23:45:31.206 UTC [32738] LOG: background worker "logical replication launcher" (PID 32747) exited with exit code 1
2026-09-28 23:45:31.206 UTC [32742] LOG: shutting down
2026-09-28 23:45:31.404 UTC [32738] LOG: database system is shut down
2026-09-28T23:45:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49437 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:45:31 dist-test-slave-c3s8 krb5kdc[1255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:45:31 dist-test-slave-c3s8 krb5kdc[1255](info): set up 2 sockets
Sep 28 23:45:31 dist-test-slave-c3s8 krb5kdc[1255](info): commencing operation
krb5kdc: starting...
W20260928 23:45:32.626242 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:45:32 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:45:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/rangeradmin_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/rangeradmin_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/rangerlookup_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/rangerlookup_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
I20260928 23:45:32.808218 21953 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzqclLI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/postgres -l logfile start
2026-09-28 23:45:35.567 UTC [1285] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:45:35.567 UTC [1285] LOG: listening on IPv4 address "127.21.112.84", port 59657
2026-09-28 23:45:35.570 UTC [1285] LOG: listening on Unix socket "/tmp/.s.PGSQL.59657"
2026-09-28 23:45:35.593 UTC [1288] LOG: database system was shut down at 2026-09-28 23:45:33 UTC
2026-09-28 23:45:35.597 UTC [1285] LOG: database system is ready to accept connections
I20260928 23:45:36.562075 21953 mini_postgres.cc:96] Postgres bound to 59657
2026-09-28 23:45:36.568 UTC [1296] FATAL: database "slave" does not exist
127.21.112.84:59657 - accepting connections
I20260928 23:45:36.569257 21953 mini_ranger.cc:161] Starting Ranger...
I20260928 23:45:36.584697 21953 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:45:37.143215 21953 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:45:37.143308 21953 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-admin
2026-09-28 23:45:37,428 [I] DB FLAVOR :POSTGRES
2026-09-28 23:45:37,429 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:45:37,429 [I] Checking connection..
2026-09-28 23:45:37,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:45:37,692 [I] Checking connection passed.
2026-09-28 23:45:37,692 [I] --------- Verifying version history table ---------
2026-09-28 23:45:37,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:45:37,968 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:45:37,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:45:38,212 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:45:38,212 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:45:38,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:45:38.446 UTC [1360] WARNING: there is no transaction in progress
2026-09-28 23:45:38,453 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:45:38,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:45:38,697 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:45:38,698 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:45:38,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:45:38,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:45:39,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c3s8.c.gcp-upstream.internal','N') ;"
2026-09-28 23:45:39,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:45:39,649 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:45:39,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:45:39,902 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:45:39,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:45:40,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:45:40,423 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:45:40,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:45:41.512 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.530 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.549 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.567 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.589 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.612 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.682 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.689 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.705 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.716 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.728 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.743 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.754 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.762 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.770 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:41.778 UTC [1468] WARNING: there is no transaction in progress
2026-09-28 23:45:42,560 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:45:42,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:45:42,802 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:45:42,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:45:43,060 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:45:43,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:45:43,309 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:45:43,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:45:43,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:45:43,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c3s8.c.gcp-upstream.internal';"
2026-09-28 23:45:44,041 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:45:44,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:45:44,292 [I] DB_PATCHES have already been applied
I20260928 23:45:44.297797 21953 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzqclLI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:45:44.297865 21953 mini_ranger.cc:193] Using host: 127.21.112.84
I20260928 23:45:44.301400 21953 mini_ranger.cc:227] Ranger admin URL: http://127.21.112.84:60927
Sep 28, 2026 11:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:45:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:45:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:45:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:45:45.413405 21953 mini_ranger.cc:160] Time spent starting Ranger: real 8.844s user 0.000s sys 0.011s
Sep 28, 2026 11:46:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:46:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 11:46:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:46:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:46:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:46:06.826349 21953 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.112.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.112.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/rangerkms_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.112.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/rangerkms_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.112.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.112.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.112.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/HTTP_127.21.112.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:46:06 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639166, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:46:06.912446 21953 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:46:06.922631 21953 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:46:07.864241 21953 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:46:07.864346 21953 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms
I20260928 23:46:07.864365 21953 mini_ranger_kms.cc:227] Using postgres at 127.21.112.84:59657
2026-09-28 23:46:08,040 [I] DB FLAVOR :POSTGRES
2026-09-28 23:46:08,040 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:46:08,040 [I] Checking connection
2026-09-28 23:46:08,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:46:08,301 [I] connection success
2026-09-28 23:46:08,301 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:46:08,301 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:46:08,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:46:08,564 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:46:08,565 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:46:08,565 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:46:08,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.112.84:59657/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:46:08,829 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:46:09.349924 21953 mini_ranger_kms.cc:310] Created kms service
I20260928 23:46:09.553692 21953 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:46:09.602164 21953 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:46:09 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639166, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:09.978976 21953 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:46:09.979147 21953 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzqclLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzqclLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:46:09.979172 21953 mini_ranger_kms.cc:240] Using host: 127.21.112.84
I20260928 23:46:09.983409 21953 mini_ranger_kms.cc:276] Ranger KMS PID: 1648
I20260928 23:46:09.983506 21953 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.112.84:33153
Sep 28, 2026 11:46:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:46:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:46:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:46:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:46:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:46:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 23:46:12 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639172, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.112.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:12.322538 21953 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.410s user 0.017s sys 0.139s
I20260928 23:46:12.322674 21953 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:46:12.322762 21953 mini_ranger_kms.cc:401] 127.21.112.84:33153/kms/v1/keys
Sep 28 23:46:12 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639172, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.112.84@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
Sep 28 23:46:13 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639172, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.112.84@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
Sep 28 23:46:15 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:46:15.377710 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37861
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:37861
--ranger_config_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:15.494302 1699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:15.494589 1699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:15.494629 1699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:15.498780 1699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:15.498845 1699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:15.498865 1699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:15.498881 1699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:15.498898 1699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:15.504778 1699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.112.84:33153/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:37861
--ranger_config_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:15.506039 1699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:15.507128 1699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:15.516844 1704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:15.516853 1699 server_base.cc:1034] running on GCE node
W20260928 23:46:15.517122 1705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:15.517308 1707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:15.517999 1699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:15.519263 1699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:15.520464 1699 hybrid_clock.cc:648] HybridClock initialized: now 1790639175520397 us; error 103 us; skew 500 ppm
Sep 28 23:46:15 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:15.530313 1699 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:15.531527 1699 webserver.cc:460] Webserver started at http://127.21.112.126:41543/ using document root <none> and password file <none>
I20260928 23:46:15.532140 1699 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:15.532222 1699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:15.532451 1699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:46:15 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639175, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:15.590020 1699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "7bd0841346ca4c48bc404aab246115ad"
format_stamp: "Formatted at 2026-09-28 23:46:15 on dist-test-slave-c3s8"
server_key: "89064c73c15296829867a64acf14bcbe"
server_key_iv: "b833953a57772002b252be23ef57f66b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:15.590816 1699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "7bd0841346ca4c48bc404aab246115ad"
format_stamp: "Formatted at 2026-09-28 23:46:15 on dist-test-slave-c3s8"
server_key: "89064c73c15296829867a64acf14bcbe"
server_key_iv: "b833953a57772002b252be23ef57f66b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:15.595846 1699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 23:46:15.614202 1714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:15.615947 1699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:46:15.616093 1699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "7bd0841346ca4c48bc404aab246115ad"
format_stamp: "Formatted at 2026-09-28 23:46:15 on dist-test-slave-c3s8"
server_key: "89064c73c15296829867a64acf14bcbe"
server_key_iv: "b833953a57772002b252be23ef57f66b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:15.616236 1699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:15.641858 1699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:15.645527 1699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:15.645838 1699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:15.655684 1699 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:37861
I20260928 23:46:15.655702 1766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:37861 every 8 connection(s)
I20260928 23:46:15.656899 1699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:15.660293 1767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:15.666515 1767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Bootstrap starting.
I20260928 23:46:15.666530 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 1699
I20260928 23:46:15.666664 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:46:15.666970 21953 external_mini_cluster.cc:1428] Setting key a32c6659eb78bca8b24d8c60e53e9694
I20260928 23:46:15.669457 1767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:15.670387 1767 log.cc:824] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:15.673053 1767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: No bootstrap required, opened a new log
I20260928 23:46:15.677315 1767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:15.677676 1767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:15.677784 1767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd0841346ca4c48bc404aab246115ad, State: Initialized, Role: FOLLOWER
Sep 28 23:46:15 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:15.678372 1767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:15.678678 1767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:15.678777 1767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:15.678898 1767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:15.680089 1767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:15.680480 1767 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd0841346ca4c48bc404aab246115ad; no voters:
I20260928 23:46:15.680874 1767 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:46:15.680959 1772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:15.681227 1772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 LEADER]: Becoming Leader. State: Replica: 7bd0841346ca4c48bc404aab246115ad, State: Running, Role: LEADER
I20260928 23:46:15.681612 1772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:15.682375 1767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:15.682945 1772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd0841346ca4c48bc404aab246115ad. Latest consensus state: current_term: 1 leader_uuid: "7bd0841346ca4c48bc404aab246115ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } } }
I20260928 23:46:15.683141 1772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:15.683395 1772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd0841346ca4c48bc404aab246115ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } } }
I20260928 23:46:15.683715 1772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:15.684055 1779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:15.685204 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:15.668528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.21.112.126:37861)
0928 23:46:15.669031 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:15.669049 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:15.669090 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:46:15.670002 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:15.670027 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:15.670465 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:15.670855 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:15.671720 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:15.673126 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:15.674458 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:15.675198 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:15.675365 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:15.678654 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:15.678661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:15.678682 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:15.678720 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:15.681001 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:15.681665 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:15.681672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:15.681679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:15.681822 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:15.682272 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:15.682275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:15.682277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:15.682832 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:15.683666 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:15.683839 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:46:15.684152 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":157,"threads_started":1}
I20260928 23:46:15.689037 1779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:15.689414 1767 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:46:15.701891 1779 catalog_manager.cc:1347] Generated new cluster ID: 6866225fdac4466687ae189e6fd35e81
I20260928 23:46:15.702004 1779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:15.717324 1779 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:46:15.718807 1779 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:15.728042 1779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Generated new TSK 0
I20260928 23:46:15.729126 1779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:46:16.461784 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37861
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:16.582983 1812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:16.583343 1812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:16.583422 1812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:16.588008 1812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:16.588114 1812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:16.588215 1812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:46:16.594425 1812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.112.84:33153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:16.595664 1812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:16.596773 1812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:16.607209 1817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:16.607209 1818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:16.607458 1812 server_base.cc:1034] running on GCE node
W20260928 23:46:16.607616 1820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:16.608089 1812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:16.608758 1812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:16.609973 1812 hybrid_clock.cc:648] HybridClock initialized: now 1790639176609881 us; error 109 us; skew 500 ppm
Sep 28 23:46:16 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:16.612984 1812 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:46:16.614043 1812 webserver.cc:460] Webserver started at http://127.21.112.65:45709/ using document root <none> and password file <none>
I20260928 23:46:16.614781 1812 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:16.614909 1812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:16.615183 1812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:46:16 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:16.631951 1812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "de229d7d0fb74566b8a2693525cba2df"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "83be96244e47fd89ff5e7c06189f126c"
server_key_iv: "e33d8507497838f670287aa5d2d644e6"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:16.632872 1812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "de229d7d0fb74566b8a2693525cba2df"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "83be96244e47fd89ff5e7c06189f126c"
server_key_iv: "e33d8507497838f670287aa5d2d644e6"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:16.640388 1812 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260928 23:46:16.670040 1829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:16.672204 1812 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260928 23:46:16.672358 1812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "de229d7d0fb74566b8a2693525cba2df"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "83be96244e47fd89ff5e7c06189f126c"
server_key_iv: "e33d8507497838f670287aa5d2d644e6"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:16.672499 1812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:16.693950 1812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:16.697693 1812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:16.697912 1812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:16.698814 1812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:16.700167 1812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:16.700237 1812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:16.700309 1812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:16.700346 1812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:16.711216 1812 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:42313
I20260928 23:46:16.712461 1812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:46:16.720386 1942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:42313 every 8 connection(s)
I20260928 23:46:16.720882 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 1812
I20260928 23:46:16.721006 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:46:16.721306 21953 external_mini_cluster.cc:1428] Setting key a994bc0e646dd7a3d574562c32b53846
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Sep 28 23:46:16 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:46:16.747455 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:16.722956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37861 (local address 127.21.112.65:47619)
0928 23:46:16.726786 (+ 3830us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:16.726857 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:16.727661 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:16.728765 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:16.728778 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:16.729234 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:16.730040 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:16.730064 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:16.731538 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:16.731542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:16.733013 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:16.733026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:16.733862 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:16.733866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:16.734113 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:16.734873 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:16.734894 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:16.739183 (+ 4289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:16.743688 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:16.743697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:16.743720 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:16.744196 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:16.744598 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:16.744601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:16.744603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:16.744833 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:16.745358 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:16.745366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:16.745776 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 23:46:16.745992 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 23:46:16.746366 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3536,"thread_start_us":128,"threads_started":1}
I20260928 23:46:16.748337 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:16.722379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:47619 (local address 127.21.112.126:37861)
0928 23:46:16.728275 (+ 5896us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:16.728288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:16.728309 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:46:16.728360 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:16.728363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:16.728413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:16.728544 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:16.730375 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:16.731360 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:16.733178 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:16.733562 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:16.733669 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:16.739466 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:16.739471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:16.739474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:16.739517 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:16.741520 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:16.744350 (+ 2830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:16.744354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:16.744360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:16.744477 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:16.744952 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:16.744956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:16.744957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:16.745211 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:16.745303 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:16.748025 (+ 2722us) server_negotiation.cc:299] Negotiation successful
0928 23:46:16.748163 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5589,"thread_start_us":148,"threads_started":1}
I20260928 23:46:16.750186 1943 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37861
I20260928 23:46:16.750433 1943 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:16.751031 1943 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:16.752882 1727 ts_manager.cc:194] Registered new tserver with Master: de229d7d0fb74566b8a2693525cba2df (127.21.112.65:42313)
I20260928 23:46:16.754421 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:47619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:46:16.776980 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37861
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:46:16 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:46:16.942123 1951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:16.942652 1951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:16.942791 1951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:16.949623 1951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:16.949724 1951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:16.949821 1951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:46:16.957332 1951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.112.84:33153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:16.958619 1951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:16.959765 1951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:16.973588 1958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:16.973605 1960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:16.974078 1957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:16.974193 1951 server_base.cc:1034] running on GCE node
I20260928 23:46:16.974862 1951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:16.975805 1951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:16.976999 1951 hybrid_clock.cc:648] HybridClock initialized: now 1790639176976973 us; error 57 us; skew 500 ppm
Sep 28 23:46:16 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:16.981078 1951 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:46:16.982410 1951 webserver.cc:460] Webserver started at http://127.21.112.66:36387/ using document root <none> and password file <none>
I20260928 23:46:16.983255 1951 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:16.983359 1951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:16.983693 1951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:46:16 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:17.007159 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "9dfefdab3eb942dd89bd71b948f59cdf"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "7ee9e8a359ce606353154e8d201118a9"
server_key_iv: "d707e19af195d58745d51bc7bff5cdf1"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.008906 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "9dfefdab3eb942dd89bd71b948f59cdf"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "7ee9e8a359ce606353154e8d201118a9"
server_key_iv: "d707e19af195d58745d51bc7bff5cdf1"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.015249 1951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 23:46:17.044654 1967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.046798 1951 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260928 23:46:17.046973 1951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "9dfefdab3eb942dd89bd71b948f59cdf"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "7ee9e8a359ce606353154e8d201118a9"
server_key_iv: "d707e19af195d58745d51bc7bff5cdf1"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.047160 1951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:17.068347 1951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:17.076853 1951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:17.077176 1951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:17.078109 1951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:17.079272 1951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:17.079336 1951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.079506 1951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:17.079546 1951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.095523 1951 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:40921
I20260928 23:46:17.096843 1951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:46:17.097832 2080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:40921 every 8 connection(s)
I20260928 23:46:17.100091 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 1951
I20260928 23:46:17.100258 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:46:17.100615 21953 external_mini_cluster.cc:1428] Setting key 54c3c28973e44a49793f64a70a3b3283
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:46:17.118187 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:17.102093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:37795 (local address 127.21.112.126:37861)
0928 23:46:17.102267 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:17.102273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:17.103321 (+ 1048us) server_negotiation.cc:407] Connection header received
0928 23:46:17.104631 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:17.104635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:17.104711 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:17.104769 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:17.106703 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.107499 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.109300 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.109663 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.109749 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:17.113140 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:17.113145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:17.113148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:17.113182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:17.115203 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:17.116020 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:17.116024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:17.116026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:17.116134 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:17.116682 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:17.116686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:17.116688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:17.116939 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:17.117051 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:17.117843 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 23:46:17.118004 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:46:17.120458 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:17.102438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37861 (local address 127.21.112.66:37795)
0928 23:46:17.103147 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:17.103218 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:17.104350 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:17.105301 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:17.105311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:17.105748 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:17.106506 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:17.106528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.107678 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.107682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:17.109147 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:17.109159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.109991 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.109994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:17.110222 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:17.110990 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:17.111014 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:17.112868 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:17.115414 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:17.115423 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:17.115445 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:17.115877 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:17.116258 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:17.116262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:17.116264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:17.116552 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:17.117067 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:17.117075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:17.117475 (+ 400us) client_negotiation.cc:769] Sending connection context
0928 23:46:17.118474 (+ 999us) client_negotiation.cc:240] Negotiation successful
0928 23:46:17.119232 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":194,"threads_started":1}
I20260928 23:46:17.122083 2081 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37861
I20260928 23:46:17.122408 2081 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:17.123111 2081 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:17.128189 1727 ts_manager.cc:194] Registered new tserver with Master: 9dfefdab3eb942dd89bd71b948f59cdf (127.21.112.66:40921)
I20260928 23:46:17.129184 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:37795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:46:17.161423 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37861
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:17.342895 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:17.343276 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:17.343358 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:17.350049 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:17.350147 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:17.350248 2086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639176, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:17.359251 2086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.112.84:33153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:17.360682 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:17.362035 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:17.377414 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:17.377422 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:17.381817 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:17.383967 2086 server_base.cc:1034] running on GCE node
I20260928 23:46:17.384574 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:17.385456 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:17.417060 2086 hybrid_clock.cc:648] HybridClock initialized: now 1790639177416878 us; error 206 us; skew 500 ppm
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:17.421790 2086 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:46:17.423305 2086 webserver.cc:460] Webserver started at http://127.21.112.67:45477/ using document root <none> and password file <none>
I20260928 23:46:17.424193 2086 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:17.424358 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:17.424677 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:17.450717 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4"
format_stamp: "Formatted at 2026-09-28 23:46:17 on dist-test-slave-c3s8"
server_key: "cb48a083b5d945efb7da5036a7e20395"
server_key_iv: "17c3166a3bda04ce9ba65c3d44ccbb0b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.452829 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4"
format_stamp: "Formatted at 2026-09-28 23:46:17 on dist-test-slave-c3s8"
server_key: "cb48a083b5d945efb7da5036a7e20395"
server_key_iv: "17c3166a3bda04ce9ba65c3d44ccbb0b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.467383 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.001s sys 0.004s
I20260928 23:46:17.501833 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.503775 2086 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260928 23:46:17.503937 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4"
format_stamp: "Formatted at 2026-09-28 23:46:17 on dist-test-slave-c3s8"
server_key: "cb48a083b5d945efb7da5036a7e20395"
server_key_iv: "17c3166a3bda04ce9ba65c3d44ccbb0b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.504107 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:17.542234 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:17.546185 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:17.546412 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:17.547129 2086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:17.548172 2086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:17.548235 2086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.548290 2086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:17.548317 2086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.558784 2086 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:37817
I20260928 23:46:17.559212 2214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:37817 every 8 connection(s)
I20260928 23:46:17.559980 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:46:17.568022 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2086
I20260928 23:46:17.568202 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:46:17.568542 21953 external_mini_cluster.cc:1428] Setting key e1628aa99ff36fc59df07a1c8dc829bf
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:17.584154 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:17.562658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:50989 (local address 127.21.112.126:37861)
0928 23:46:17.562916 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:17.562921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:17.564591 (+ 1670us) server_negotiation.cc:407] Connection header received
0928 23:46:17.565966 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:17.565970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:17.566036 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:17.566134 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:17.567769 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.568562 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.570954 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.571359 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.571457 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:17.575730 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:17.575734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:17.575738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:17.575771 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:17.579346 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:17.580146 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:17.580151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:17.580153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:17.580245 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:17.580846 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:17.580850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:17.580852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:17.581197 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:17.581309 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:17.583827 (+ 2518us) server_negotiation.cc:299] Negotiation successful
0928 23:46:17.583962 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 23:46:17.585903 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:17.563253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37861 (local address 127.21.112.67:50989)
0928 23:46:17.564352 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:17.564418 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:17.565645 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:17.566373 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:17.566384 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:17.566798 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:17.567565 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:17.567589 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.569092 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.569096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:17.570770 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:17.570784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:17.571702 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:17.571706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:17.571942 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:17.573742 (+ 1800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:17.573764 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:17.575483 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:17.579569 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:17.579577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:17.579597 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:17.580000 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:17.580362 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:17.580365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:17.580367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:17.580684 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:17.581407 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:17.581439 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:17.583525 (+ 2086us) client_negotiation.cc:769] Sending connection context
0928 23:46:17.584367 (+ 842us) client_negotiation.cc:240] Negotiation successful
0928 23:46:17.584754 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":159,"threads_started":1}
I20260928 23:46:17.587150 2215 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37861
I20260928 23:46:17.587412 2215 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:17.588061 2215 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:17.589252 1727 ts_manager.cc:194] Registered new tserver with Master: d33424c3c2ca4fe3b63dcdc92d6517a4 (127.21.112.67:37817)
I20260928 23:46:17.590078 1727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:50989
I20260928 23:46:17.595063 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:46:17.595885 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 1812
I20260928 23:46:17.605521 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 1951
I20260928 23:46:17.612417 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2086
I20260928 23:46:17.620527 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 1699
I20260928 23:46:17.628804 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37861
--webserver_interface=127.21.112.126
--webserver_port=41543
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:37861
--ranger_config_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:17.746240 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:17.746548 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:17.746592 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:17.750898 2218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:17.750975 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:17.750995 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:17.751011 2218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:17.751027 2218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:17.757050 2218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:37861
--ranger_config_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:17.758301 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:17.759393 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:17.768543 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:17.768543 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:17.768813 2218 server_base.cc:1034] running on GCE node
W20260928 23:46:17.769011 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:17.769711 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:17.770887 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:17.772117 2218 hybrid_clock.cc:648] HybridClock initialized: now 1790639177772074 us; error 57 us; skew 500 ppm
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:17.775272 2218 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:17.776315 2218 webserver.cc:460] Webserver started at http://127.21.112.126:41543/ using document root <none> and password file <none>
I20260928 23:46:17.776929 2218 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:17.777032 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:46:17 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639177, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:17.834874 2218 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:46:17.838073 2234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:17.839910 2218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:46:17.840052 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "7bd0841346ca4c48bc404aab246115ad"
format_stamp: "Formatted at 2026-09-28 23:46:15 on dist-test-slave-c3s8"
server_key: "89064c73c15296829867a64acf14bcbe"
server_key_iv: "b833953a57772002b252be23ef57f66b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:17.840746 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:17.856801 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:17.860029 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:17.860204 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:17.867753 2218 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:37861
I20260928 23:46:17.867767 2286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:37861 every 8 connection(s)
I20260928 23:46:17.868852 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:17.873389 2287 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:46:17.875231 2287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Bootstrap starting.
I20260928 23:46:17.877796 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2218
I20260928 23:46:17.878328 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:42313
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:37861
--webserver_port=45709
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:17.893088 2287 log.cc:824] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:17.899996 2287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:46:17.900525 2287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Bootstrap complete.
I20260928 23:46:17.904927 2287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:17.905371 2287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd0841346ca4c48bc404aab246115ad, State: Initialized, Role: FOLLOWER
I20260928 23:46:17.905941 2287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:17.906097 2287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:17.906186 2287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:17.906304 2287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:46:17.908424 2287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:17.908778 2287 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd0841346ca4c48bc404aab246115ad; no voters:
I20260928 23:46:17.909138 2287 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:46:17.909252 2291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:46:17.909453 2291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [term 2 LEADER]: Becoming Leader. State: Replica: 7bd0841346ca4c48bc404aab246115ad, State: Running, Role: LEADER
I20260928 23:46:17.909792 2291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } }
I20260928 23:46:17.910461 2287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:17.911187 2292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7bd0841346ca4c48bc404aab246115ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } } }
I20260928 23:46:17.911324 2292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:17.911963 2293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd0841346ca4c48bc404aab246115ad. Latest consensus state: current_term: 2 leader_uuid: "7bd0841346ca4c48bc404aab246115ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd0841346ca4c48bc404aab246115ad" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37861 } } }
I20260928 23:46:17.912127 2293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:17.912376 2300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:17.912813 2287 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:46:17.919442 2300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:17.920543 2300 catalog_manager.cc:1259] Loaded cluster ID: 6866225fdac4466687ae189e6fd35e81
I20260928 23:46:17.920616 2300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:17.923532 2300 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:17.924348 2300 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Loaded TSK: 0
I20260928 23:46:17.925822 2300 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:46:18.047801 2289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:18.048202 2289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:18.048272 2289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:18.054728 2289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:18.054828 2289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:18.054922 2289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:46:18.063506 2289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:42313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=45709
--tserver_master_addrs=127.21.112.126:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:18.065066 2289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:18.066622 2289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:18.079355 2325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:18.079640 2327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:18.080019 2324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:18.080142 2289 server_base.cc:1034] running on GCE node
I20260928 23:46:18.080613 2289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:18.081386 2289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:18.082547 2289 hybrid_clock.cc:648] HybridClock initialized: now 1790639178082507 us; error 57 us; skew 500 ppm
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:18.086052 2289 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:46:18.087250 2289 webserver.cc:460] Webserver started at http://127.21.112.65:45709/ using document root <none> and password file <none>
I20260928 23:46:18.088012 2289 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:18.088135 2289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:18.115011 2289 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 23:46:18.122517 2335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.124725 2289 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260928 23:46:18.124898 2289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "de229d7d0fb74566b8a2693525cba2df"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "83be96244e47fd89ff5e7c06189f126c"
server_key_iv: "e33d8507497838f670287aa5d2d644e6"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:18.125766 2289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:18.156189 2289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:18.159723 2289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:18.159950 2289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:18.160625 2289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:18.161631 2289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:18.161698 2289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.161762 2289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:18.161792 2289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.173622 2289 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:42313
I20260928 23:46:18.173889 2448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:42313 every 8 connection(s)
I20260928 23:46:18.175006 2289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:46:18.182361 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2289
I20260928 23:46:18.182945 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:40921
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:37861
--webserver_port=36387
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:18.197110 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.177727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:57379 (local address 127.21.112.126:37861)
0928 23:46:18.178359 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:18.178381 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:18.178861 (+ 480us) server_negotiation.cc:407] Connection header received
0928 23:46:18.180424 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:18.180443 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:18.180867 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:18.181214 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:18.183115 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.184720 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.186436 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.187206 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.187350 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.190661 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:18.190669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:18.190689 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:18.190724 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:18.192865 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.193597 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.193604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.193611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.193830 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.194315 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.194321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.194330 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.194903 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:18.195045 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:18.195826 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 23:46:18.196095 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":132,"threads_started":1}
I20260928 23:46:18.197110 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.177936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37861 (local address 127.21.112.65:57379)
0928 23:46:18.178676 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:18.178732 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:18.179874 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:18.181843 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:18.181852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:18.182245 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:18.182903 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:18.182922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.184896 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.184899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:18.186284 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:18.186296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.187637 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.187642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:18.187853 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.188606 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:18.188630 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:18.190403 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:18.193029 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:18.193036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:18.193054 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:18.193414 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:18.193979 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:18.193982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:18.193984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:18.194180 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:18.195106 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:18.195114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:18.195435 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 23:46:18.195677 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 23:46:18.196049 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":165,"threads_started":1}
I20260928 23:46:18.199661 2449 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37861
I20260928 23:46:18.199905 2449 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.200380 2449 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
W20260928 23:46:18.202521 2449 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:37861 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:46:18.202667 2449 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.202854 2449 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:18.203763 2449 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.203966 2449 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
W20260928 23:46:18.204789 2449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:46:18.326913 2453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:18.327288 2453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:18.327358 2453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:18.332471 2453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:18.332537 2453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:18.332599 2453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:46:18.338546 2453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:40921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=36387
--tserver_master_addrs=127.21.112.126:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:18.339936 2453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:18.341233 2453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:18.353230 2459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:18.354148 2458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:18.354622 2461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:18.354799 2453 server_base.cc:1034] running on GCE node
I20260928 23:46:18.355299 2453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:18.356158 2453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:18.358963 2453 hybrid_clock.cc:648] HybridClock initialized: now 1790639178358917 us; error 61 us; skew 500 ppm
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:18.362782 2453 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:46:18.364037 2453 webserver.cc:460] Webserver started at http://127.21.112.66:36387/ using document root <none> and password file <none>
I20260928 23:46:18.364848 2453 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:18.365026 2453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:18.390527 2453 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260928 23:46:18.394071 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.395897 2453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:46:18.396031 2453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "9dfefdab3eb942dd89bd71b948f59cdf"
format_stamp: "Formatted at 2026-09-28 23:46:16 on dist-test-slave-c3s8"
server_key: "7ee9e8a359ce606353154e8d201118a9"
server_key_iv: "d707e19af195d58745d51bc7bff5cdf1"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:18.396850 2453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:18.414788 2453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:18.418915 2453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:18.419143 2453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:18.419880 2453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:18.420956 2453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:18.421006 2453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.421058 2453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:18.421080 2453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.433889 2453 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:40921
I20260928 23:46:18.434029 2582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:40921 every 8 connection(s)
I20260928 23:46:18.435395 2453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:46:18.444134 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2453
I20260928 23:46:18.444711 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.112.84:33153/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:37817
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:37861
--webserver_port=45477
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:18.454144 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.438687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:49229 (local address 127.21.112.126:37861)
0928 23:46:18.438848 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:18.438853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:18.439816 (+ 963us) server_negotiation.cc:407] Connection header received
0928 23:46:18.441153 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:18.441157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:18.441215 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:18.441272 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:18.442924 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.443786 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.445834 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.446227 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.446308 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.449367 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:18.449372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:18.449375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:18.449404 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:18.451313 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.452029 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.452033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.452035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.452130 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.452586 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.452589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.452591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.452850 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:18.452924 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:18.453704 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 23:46:18.453870 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:46:18.455452 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.439139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37861 (local address 127.21.112.66:49229)
0928 23:46:18.439610 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:18.439713 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:18.440885 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:18.441719 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:18.441729 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:18.442081 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:18.442745 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:18.442763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.443997 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.444000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:18.445692 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:18.445704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.446545 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.446548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:18.446742 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.447427 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:18.447448 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:18.449149 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:18.451457 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:18.451466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:18.451487 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:18.451882 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:18.452245 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:18.452248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:18.452250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:18.452453 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:18.453007 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:18.453014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:18.453347 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 23:46:18.453648 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 23:46:18.454026 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":138,"threads_started":1}
I20260928 23:46:18.457001 2584 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37861
I20260928 23:46:18.457280 2584 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.457963 2584 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
W20260928 23:46:18.459880 2584 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:37861 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:46:18.460022 2584 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.460229 2584 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:18.461169 2584 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.461359 2584 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
W20260928 23:46:18.462203 2584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:46:18.579152 2587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:18.579523 2587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:18.579595 2587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:18.585749 2587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:18.585821 2587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:18.585906 2587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:46:18.593859 2587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.112.84:33153/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:37817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=45477
--tserver_master_addrs=127.21.112.126:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:18.595273 2587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:18.596556 2587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:18.608820 2593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:18.609375 2595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:18.609526 2592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:18.613631 2587 server_base.cc:1034] running on GCE node
I20260928 23:46:18.614290 2587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:18.615134 2587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:18.622515 2587 hybrid_clock.cc:648] HybridClock initialized: now 1790639178622438 us; error 93 us; skew 500 ppm
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:18.625649 2587 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:46:18.626735 2587 webserver.cc:460] Webserver started at http://127.21.112.67:45477/ using document root <none> and password file <none>
I20260928 23:46:18.627444 2587 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:18.627545 2587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:18.659178 2587 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260928 23:46:18.662645 2603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.664193 2587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:46:18.664340 2587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4"
format_stamp: "Formatted at 2026-09-28 23:46:17 on dist-test-slave-c3s8"
server_key: "cb48a083b5d945efb7da5036a7e20395"
server_key_iv: "17c3166a3bda04ce9ba65c3d44ccbb0b"
server_key_version: "kuduclusterkey@0"
I20260928 23:46:18.664947 2587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:18.712827 2587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:18.722507 2587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:18.722740 2587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:18.723418 2587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:18.724351 2587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:18.724417 2587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.724483 2587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:18.724514 2587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:18.734699 2587 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:37817
I20260928 23:46:18.736017 2587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:46:18.739693 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2587
I20260928 23:46:18.743330 2716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:37817 every 8 connection(s)
I20260928 23:46:18.752940 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.740483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.21.112.126:37861)
0928 23:46:18.740627 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:18.740632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:18.740930 (+ 298us) server_negotiation.cc:407] Connection header received
0928 23:46:18.741183 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:18.741188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:18.741254 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:18.741320 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:18.742630 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.743598 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.745264 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.745709 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.745810 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.748121 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:18.748127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:18.748132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:18.748173 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:18.750362 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.751050 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.751055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.751059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.751161 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.751521 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.751528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.751531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.751842 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:18.752035 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:18.752507 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 23:46:18.752692 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":51}
Sep 28 23:46:18 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639178, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:18.767720 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.746191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:37861 (local address 127.21.112.67:36039)
0928 23:46:18.747807 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:18.747864 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:18.748921 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:18.749900 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:18.749911 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:18.750285 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:18.751032 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:18.751056 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.752469 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.752473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:18.754023 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:18.754036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.755157 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.755161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:18.755448 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.756211 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:18.756234 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:18.759226 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:18.761893 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:18.761901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:18.761924 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:18.762385 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:18.762759 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:18.762763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:18.762765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:18.762986 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:18.763536 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:18.763543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:18.765802 (+ 2259us) client_negotiation.cc:769] Sending connection context
0928 23:46:18.766093 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 23:46:18.766519 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"thread_start_us":196,"threads_started":1}
I20260928 23:46:18.768795 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:18.747378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:36039 (local address 127.21.112.126:37861)
0928 23:46:18.749354 (+ 1976us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:18.749360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:18.749374 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:46:18.749413 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:18.749417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:18.749465 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:18.749577 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:18.751359 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.752294 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.754477 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:18.754840 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:18.754927 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:18.759478 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:18.759484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:18.759489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:18.759529 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:18.761730 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.762528 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.762532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.762534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.762641 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:18.763109 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:18.763113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:18.763116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:18.763405 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:18.763514 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:18.768421 (+ 4907us) server_negotiation.cc:299] Negotiation successful
0928 23:46:18.768602 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":1897,"thread_start_us":92,"threads_started":1}
I20260928 23:46:18.770159 2717 heartbeater.cc:347] Connected to a master server at 127.21.112.126:37861
I20260928 23:46:18.770435 2717 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.771075 2717 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
W20260928 23:46:18.772984 2717 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:37861 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:46:18.773123 2717 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.773334 2717 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:18.774263 2717 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:18.774448 2717 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
W20260928 23:46:18.775238 2717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:46:19 dist-test-slave-c3s8 krb5kdc[1255](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790639179, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:19.205044 2449 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:19.205359 2449 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:19.206625 2250 ts_manager.cc:194] Registered new tserver with Master: de229d7d0fb74566b8a2693525cba2df (127.21.112.65:42313)
I20260928 23:46:19.208282 2250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:57379
I20260928 23:46:19.462456 2584 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:19.462790 2584 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:19.463657 2250 ts_manager.cc:194] Registered new tserver with Master: 9dfefdab3eb942dd89bd71b948f59cdf (127.21.112.66:40921)
I20260928 23:46:19.464442 2250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:49229
Sep 28 23:46:19 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639179, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
Sep 28 23:46:19 dist-test-slave-c3s8 krb5kdc[1255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639179, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for HTTP/127.21.112.84@KRBTEST.COM
I20260928 23:46:19.775472 2717 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:19.775815 2717 heartbeater.cc:511] Master 127.21.112.126:37861 requested a full tablet report, sending...
I20260928 23:46:19.776558 2251 ts_manager.cc:194] Registered new tserver with Master: d33424c3c2ca4fe3b63dcdc92d6517a4 (127.21.112.67:37817)
I20260928 23:46:19.777359 2251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:36039
I20260928 23:46:19.778227 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:46:19.800392 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.21.112.126:37861)
0928 23:46:19.788926 (+ 2872us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:19.788933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:19.788951 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:46:19.789016 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:19.789019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:19.789071 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:19.789174 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:19.790986 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.791867 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.793155 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.793566 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.793683 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.795656 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:19.795660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:19.795663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:19.795699 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:19.797639 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:19.798203 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:19.798206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:19.798209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:19.798305 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:19.798614 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:19.798617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:19.798619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:19.798975 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:19.799113 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:19.800043 (+ 930us) server_negotiation.cc:299] Negotiation successful
0928 23:46:19.800200 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2777,"thread_start_us":101,"threads_started":1}
I20260928 23:46:19.805850 2250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:46:19.811544 2250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:46:19.850113 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.825881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37817 (local address 127.0.0.1:43942)
0928 23:46:19.837442 (+ 11561us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:19.837457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:19.837577 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:19.841107 (+ 3530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:19.841111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:19.841131 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:19.841235 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.841241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.843404 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.843407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.844822 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.844835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.849489 (+ 4654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.849493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.849644 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.849686 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:46:19.849803 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:46:19.849936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11489,"thread_start_us":78,"threads_started":1}
I20260928 23:46:19.852332 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.827068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.21.112.65:42313)
0928 23:46:19.829164 (+ 2096us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:19.829172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:19.829198 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:46:19.829279 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:19.829284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:19.829558 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:19.836246 (+ 6688us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:19.836408 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.841932 (+ 5524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.847243 (+ 5311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.848142 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.851829 (+ 3687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.852003 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:19.852075 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:46:19.852193 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1988,"thread_start_us":80,"threads_started":1}
I20260928 23:46:19.851733 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.825581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:42313 (local address 127.0.0.1:58392)
0928 23:46:19.827060 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:19.827110 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:19.827234 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:19.835860 (+ 8626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:19.835868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:19.835899 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:19.836065 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.836073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.845025 (+ 8952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.845029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.847069 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.847080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.851348 (+ 4268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.851352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.851445 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.851468 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:46:19.851516 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:46:19.851585 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"thread_start_us":102,"threads_started":1}
I20260928 23:46:19.859166 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.826372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:40921 (local address 127.0.0.1:57848)
0928 23:46:19.853023 (+ 26651us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:19.853037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:19.853116 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:19.853939 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:19.853942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:19.853959 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:19.854033 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.854038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.855464 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.855467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.857421 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.857432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.858734 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.858738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.858822 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.858845 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:46:19.858958 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 23:46:19.859029 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26590,"thread_start_us":85,"threads_started":1}
I20260928 23:46:19.860008 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.827479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.21.112.66:40921)
0928 23:46:19.835413 (+ 7934us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:19.835437 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:19.853227 (+ 17790us) server_negotiation.cc:407] Connection header received
0928 23:46:19.853321 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:19.853328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:19.853611 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:19.853763 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:19.854178 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.855335 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.857585 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.858417 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.858520 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.858676 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:19.859771 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0928 23:46:19.859878 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7788,"thread_start_us":88,"threads_started":1}
I20260928 23:46:19.862001 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.827410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.21.112.67:37817)
0928 23:46:19.827783 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:19.827790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:19.837698 (+ 9908us) server_negotiation.cc:407] Connection header received
0928 23:46:19.837790 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:19.837796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:19.840759 (+ 2963us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:19.840932 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:19.842070 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.843261 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.848298 (+ 5037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.849156 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.849262 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.849426 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:19.861737 (+ 12311us) server_negotiation.cc:299] Negotiation successful
0928 23:46:19.861866 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":90,"threads_started":1}
I20260928 23:46:19.863854 2379 tablet_service.cc:1467] Processing CreateTablet for tablet d17d636c49ca44dda668d4307428bd16 (DEFAULT_TABLE table=test-table [id=6b21b43ba33849e38f50e62551b0c50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:19.865007 2651 tablet_service.cc:1467] Processing CreateTablet for tablet d17d636c49ca44dda668d4307428bd16 (DEFAULT_TABLE table=test-table [id=6b21b43ba33849e38f50e62551b0c50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:19.866217 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17d636c49ca44dda668d4307428bd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:19.866253 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17d636c49ca44dda668d4307428bd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:19.877617 2754 tablet_bootstrap.cc:492] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: Bootstrap starting.
I20260928 23:46:19.877213 2517 tablet_service.cc:1467] Processing CreateTablet for tablet d17d636c49ca44dda668d4307428bd16 (DEFAULT_TABLE table=test-table [id=6b21b43ba33849e38f50e62551b0c50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:19.878471 2517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17d636c49ca44dda668d4307428bd16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:19.880404 2754 tablet_bootstrap.cc:654] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:19.881220 2754 log.cc:824] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:19.883432 2754 tablet_bootstrap.cc:492] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: No bootstrap required, opened a new log
I20260928 23:46:19.883634 2754 ts_tablet_manager.cc:1397] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260928 23:46:19.887822 2754 raft_consensus.cc:348] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.888106 2754 raft_consensus.cc:374] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:19.888152 2754 raft_consensus.cc:729] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de229d7d0fb74566b8a2693525cba2df, State: Initialized, Role: FOLLOWER
I20260928 23:46:19.890024 2754 consensus_queue.cc:260] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.891178 2754 ts_tablet_manager.cc:1428] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260928 23:46:19.891570 2449 heartbeater.cc:503] Master 127.21.112.126:37861 was elected leader, sending a full tablet report...
I20260928 23:46:19.895661 2756 tablet_bootstrap.cc:492] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Bootstrap starting.
I20260928 23:46:19.898350 2756 tablet_bootstrap.cc:654] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:19.898949 2753 tablet_bootstrap.cc:492] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Bootstrap starting.
I20260928 23:46:19.902626 2753 tablet_bootstrap.cc:654] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:19.903563 2753 log.cc:824] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:19.903910 2756 log.cc:824] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:19.913056 2756 tablet_bootstrap.cc:492] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: No bootstrap required, opened a new log
I20260928 23:46:19.913410 2756 ts_tablet_manager.cc:1397] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20260928 23:46:19.918072 2756 raft_consensus.cc:348] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.918536 2756 raft_consensus.cc:374] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:19.918649 2756 raft_consensus.cc:729] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfefdab3eb942dd89bd71b948f59cdf, State: Initialized, Role: FOLLOWER
I20260928 23:46:19.919278 2756 consensus_queue.cc:260] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.920589 2756 ts_tablet_manager.cc:1428] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:46:19.921080 2584 heartbeater.cc:503] Master 127.21.112.126:37861 was elected leader, sending a full tablet report...
W20260928 23:46:19.928419 2450 tablet.cc:2369] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:46:19.928732 2753 tablet_bootstrap.cc:492] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: No bootstrap required, opened a new log
I20260928 23:46:19.928959 2753 ts_tablet_manager.cc:1397] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20260928 23:46:19.933185 2753 raft_consensus.cc:348] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.933601 2753 raft_consensus.cc:374] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:19.933696 2753 raft_consensus.cc:729] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33424c3c2ca4fe3b63dcdc92d6517a4, State: Initialized, Role: FOLLOWER
I20260928 23:46:19.934319 2753 consensus_queue.cc:260] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.935464 2753 ts_tablet_manager.cc:1428] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 23:46:19.935563 2717 heartbeater.cc:503] Master 127.21.112.126:37861 was elected leader, sending a full tablet report...
W20260928 23:46:19.939631 2585 tablet.cc:2369] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:46:19.956424 2757 raft_consensus.cc:482] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:19.956672 2757 raft_consensus.cc:504] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.958137 2757 leader_election.cc:288] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dfefdab3eb942dd89bd71b948f59cdf (127.21.112.66:40921), d33424c3c2ca4fe3b63dcdc92d6517a4 (127.21.112.67:37817)
I20260928 23:46:19.965186 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.958850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:60081 (local address 127.21.112.66:40921)
0928 23:46:19.959119 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:19.959124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:19.959460 (+ 336us) server_negotiation.cc:407] Connection header received
0928 23:46:19.959673 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:19.959676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:19.959722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:19.959769 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:19.960181 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.961047 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.963858 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.964470 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.964561 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.964624 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:19.964965 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 23:46:19.965046 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 23:46:19.965843 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.958412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:40921 (local address 127.21.112.65:60081)
0928 23:46:19.959374 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:19.959387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:19.959539 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:19.959903 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:19.959906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:19.959922 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:19.960059 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.960065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.962320 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.962323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.963704 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.963716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.964724 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.964727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.964833 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.964855 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:46:19.965600 (+ 745us) client_negotiation.cc:240] Negotiation successful
0928 23:46:19.965698 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":142,"threads_started":1}
I20260928 23:46:19.966624 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17d636c49ca44dda668d4307428bd16" candidate_uuid: "de229d7d0fb74566b8a2693525cba2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" is_pre_election: true
I20260928 23:46:19.966957 2537 raft_consensus.cc:2454] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de229d7d0fb74566b8a2693525cba2df in term 0.
I20260928 23:46:19.967526 2336 leader_election.cc:302] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfefdab3eb942dd89bd71b948f59cdf, de229d7d0fb74566b8a2693525cba2df; no voters:
I20260928 23:46:19.967859 2757 raft_consensus.cc:2781] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:46:19.967923 2757 raft_consensus.cc:482] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:46:19.967960 2757 raft_consensus.cc:3037] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:19.969347 2757 raft_consensus.cc:504] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.969766 2757 leader_election.cc:288] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [CANDIDATE]: Term 1 election: Requested vote from peers 9dfefdab3eb942dd89bd71b948f59cdf (127.21.112.66:40921), d33424c3c2ca4fe3b63dcdc92d6517a4 (127.21.112.67:37817)
I20260928 23:46:19.970553 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17d636c49ca44dda668d4307428bd16" candidate_uuid: "de229d7d0fb74566b8a2693525cba2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfefdab3eb942dd89bd71b948f59cdf"
I20260928 23:46:19.970695 2537 raft_consensus.cc:3037] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:19.972033 2537 raft_consensus.cc:2454] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de229d7d0fb74566b8a2693525cba2df in term 1.
I20260928 23:46:19.972735 2336 leader_election.cc:302] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfefdab3eb942dd89bd71b948f59cdf, de229d7d0fb74566b8a2693525cba2df; no voters:
I20260928 23:46:19.972960 2757 raft_consensus.cc:2781] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:19.973171 2757 raft_consensus.cc:686] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [term 1 LEADER]: Becoming Leader. State: Replica: de229d7d0fb74566b8a2693525cba2df, State: Running, Role: LEADER
I20260928 23:46:19.973564 2757 consensus_queue.cc:237] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } }
I20260928 23:46:19.977211 2250 catalog_manager.cc:5495] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df reported cstate change: term changed from 0 to 1, leader changed from <none> to de229d7d0fb74566b8a2693525cba2df (127.21.112.65). New cstate: current_term: 1 leader_uuid: "de229d7d0fb74566b8a2693525cba2df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de229d7d0fb74566b8a2693525cba2df" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 } health_report { overall_health: UNKNOWN } } }
I20260928 23:46:19.984829 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.958959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:35667 (local address 127.21.112.67:37817)
0928 23:46:19.959210 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:19.959215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:19.972203 (+ 12988us) server_negotiation.cc:407] Connection header received
0928 23:46:19.972268 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:19.972271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:19.972323 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:19.972398 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:19.981050 (+ 8652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.981901 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.983510 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.984125 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.984219 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.984272 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:19.984615 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 23:46:19.984698 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260928 23:46:19.985423 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:19.958472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:37817 (local address 127.21.112.65:35667)
0928 23:46:19.971207 (+ 12735us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:19.971225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:19.971356 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:19.980735 (+ 9379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:19.980739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:19.980761 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:19.980893 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.980899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.982052 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.982056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.983358 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:19.983369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:19.984368 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:19.984371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:19.984472 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:19.984495 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:46:19.985185 (+ 690us) client_negotiation.cc:240] Negotiation successful
0928 23:46:19.985268 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12646,"thread_start_us":120,"threads_started":1}
I20260928 23:46:19.986390 2671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17d636c49ca44dda668d4307428bd16" candidate_uuid: "de229d7d0fb74566b8a2693525cba2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" is_pre_election: true
I20260928 23:46:19.986752 2671 raft_consensus.cc:2454] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de229d7d0fb74566b8a2693525cba2df in term 0.
I20260928 23:46:19.987136 2670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17d636c49ca44dda668d4307428bd16" candidate_uuid: "de229d7d0fb74566b8a2693525cba2df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4"
I20260928 23:46:19.987264 2670 raft_consensus.cc:3037] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:19.988659 2670 raft_consensus.cc:2454] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de229d7d0fb74566b8a2693525cba2df in term 1.
W20260928 23:46:19.995846 2718 tablet.cc:2369] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:46:20.006397 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:20.000951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.21.112.65:42313)
0928 23:46:20.001064 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:20.001069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:20.001084 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:46:20.001173 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:20.001176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:20.001241 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:20.001355 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:46:20.001824 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:20.002843 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:20.004108 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:20.004461 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:20.004554 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:20.004992 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:46:20.005605 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:46:20.005722 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:20.006159 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 23:46:20.006227 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:46:20.020845 2670 raft_consensus.cc:1261] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 1 FOLLOWER]: Refusing update from remote peer de229d7d0fb74566b8a2693525cba2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:20.021994 2764 consensus_queue.cc:1035] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33424c3c2ca4fe3b63dcdc92d6517a4" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 37817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:20.030331 2537 raft_consensus.cc:1261] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 1 FOLLOWER]: Refusing update from remote peer de229d7d0fb74566b8a2693525cba2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:20.031265 2764 consensus_queue.cc:1035] T d17d636c49ca44dda668d4307428bd16 P de229d7d0fb74566b8a2693525cba2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfefdab3eb942dd89bd71b948f59cdf" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 40921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:20.047919 2767 mvcc.cc:205] Tried to move back new op lower bound from 7334458081334181888 to 7334458081175482368. Current Snapshot: MvccSnapshot[applied={T|T < 7334458081334181888}]
I20260928 23:46:20.049577 2770 mvcc.cc:205] Tried to move back new op lower bound from 7334458081334181888 to 7334458081175482368. Current Snapshot: MvccSnapshot[applied={T|T < 7334458081334181888}]
I20260928 23:46:20.058127 2768 mvcc.cc:205] Tried to move back new op lower bound from 7334458081334181888 to 7334458081175482368. Current Snapshot: MvccSnapshot[applied={T|T < 7334458081334181888}]
I20260928 23:46:20.068547 2250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:20.068758 2250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:20.071887 2250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bd0841346ca4c48bc404aab246115ad: Sending DeleteTablet for 3 replicas of tablet d17d636c49ca44dda668d4307428bd16
I20260928 23:46:20.073061 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet d17d636c49ca44dda668d4307428bd16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:20 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:58392
I20260928 23:46:20.073670 2517 tablet_service.cc:1514] Processing DeleteTablet for tablet d17d636c49ca44dda668d4307428bd16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:20 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:57848
I20260928 23:46:20.073863 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2289
I20260928 23:46:20.073581 2651 tablet_service.cc:1514] Processing DeleteTablet for tablet d17d636c49ca44dda668d4307428bd16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:20 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:43942
I20260928 23:46:20.081745 2777 tablet_replica.cc:330] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: stopping tablet replica
I20260928 23:46:20.081938 2778 tablet_replica.cc:330] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: stopping tablet replica
I20260928 23:46:20.082104 2777 raft_consensus.cc:2229] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:20.082340 2778 raft_consensus.cc:2229] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:20.082495 2777 raft_consensus.cc:2258] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:20.082909 2778 raft_consensus.cc:2258] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:20.084754 2777 ts_tablet_manager.cc:1905] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:20.087870 2777 ts_tablet_manager.cc:1918] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:20.087984 2777 log.cc:1196] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/d17d636c49ca44dda668d4307428bd16
I20260928 23:46:20.088339 2777 ts_tablet_manager.cc:1939] T d17d636c49ca44dda668d4307428bd16 P 9dfefdab3eb942dd89bd71b948f59cdf: Deleting consensus metadata
I20260928 23:46:20.089846 2235 catalog_manager.cc:4841] TS 9dfefdab3eb942dd89bd71b948f59cdf (127.21.112.66:40921): tablet d17d636c49ca44dda668d4307428bd16 (table test-table [id=6b21b43ba33849e38f50e62551b0c50b]) successfully deleted
I20260928 23:46:20.095278 2778 ts_tablet_manager.cc:1905] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:46:20.095837 2235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:42313 (error 108)
I20260928 23:46:20.096849 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2453
I20260928 23:46:20.098297 2778 ts_tablet_manager.cc:1918] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:20.098412 2778 log.cc:1196] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/d17d636c49ca44dda668d4307428bd16
I20260928 23:46:20.098760 2778 ts_tablet_manager.cc:1939] T d17d636c49ca44dda668d4307428bd16 P d33424c3c2ca4fe3b63dcdc92d6517a4: Deleting consensus metadata
I20260928 23:46:20.102342 2236 catalog_manager.cc:4841] TS d33424c3c2ca4fe3b63dcdc92d6517a4 (127.21.112.67:37817): tablet d17d636c49ca44dda668d4307428bd16 (table test-table [id=6b21b43ba33849e38f50e62551b0c50b]) successfully deleted
I20260928 23:46:20.111099 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2587
I20260928 23:46:20.117014 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:20.116626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:42313 (local address 127.0.0.1:58400)
0928 23:46:20.116788 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:20.116896 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:42313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 23:46:20.117261 2235 catalog_manager.cc:4584] TS de229d7d0fb74566b8a2693525cba2df (127.21.112.65:42313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d17d636c49ca44dda668d4307428bd16: Network error: Client connection negotiation failed: client connection to 127.21.112.65:42313: connect: Connection refused (error 111)
I20260928 23:46:20.117677 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2218
I20260928 23:46:20.125715 21953 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:46:20.471076 21953 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:46:20.471215 21953 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:46:20.816920 21953 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:46:20.817 UTC [1285] LOG: received smart shutdown request
2026-09-28 23:46:20.821 UTC [1285] LOG: background worker "logical replication launcher" (PID 1294) exited with exit code 1
2026-09-28 23:46:20.821 UTC [1289] LOG: shutting down
2026-09-28 23:46:21.033 UTC [1285] LOG: database system is shut down
2026-09-28T23:46:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49636 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:46:21 dist-test-slave-c3s8 krb5kdc[2788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:46:21 dist-test-slave-c3s8 krb5kdc[2788](info): set up 2 sockets
Sep 28 23:46:21 dist-test-slave-c3s8 krb5kdc[2788](info): commencing operation
krb5kdc: starting...
W20260928 23:46:22.214119 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:46:22.345456 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:40939
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:40939,127.21.112.125:42793,127.21.112.124:42525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:22.459977 2801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:22.460266 2801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:22.460309 2801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:22.464920 2801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:22.465018 2801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:22.465055 2801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:22.465091 2801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:22.465128 2801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:22.470952 2801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:40939,127.21.112.125:42793,127.21.112.124:42525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:22.472144 2801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:22.473237 2801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:22.482366 2809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:22.482390 2807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:22.482726 2801 server_base.cc:1034] running on GCE node
W20260928 23:46:22.482366 2806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:22.483781 2801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:22.485035 2801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:22.486227 2801 hybrid_clock.cc:648] HybridClock initialized: now 1790639182486192 us; error 52 us; skew 500 ppm
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:22.489172 2801 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:22.490300 2801 webserver.cc:460] Webserver started at http://127.21.112.126:43163/ using document root <none> and password file <none>
I20260928 23:46:22.490932 2801 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:22.491030 2801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:22.491281 2801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:22.493467 2801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "ec14c124ad9c484085fa7f73d266fd0c"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "1eaa8d8fe9fe5e6c12512e8dba2feb83"
server_key_iv: "dee4932d2428f0640f37363f1eba5aff"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.494136 2801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "ec14c124ad9c484085fa7f73d266fd0c"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "1eaa8d8fe9fe5e6c12512e8dba2feb83"
server_key_iv: "dee4932d2428f0640f37363f1eba5aff"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.497982 2801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:46:22.500380 2816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:22.501639 2801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:46:22.501788 2801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "ec14c124ad9c484085fa7f73d266fd0c"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "1eaa8d8fe9fe5e6c12512e8dba2feb83"
server_key_iv: "dee4932d2428f0640f37363f1eba5aff"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.501902 2801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:22.515616 2801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:22.519477 2801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:22.519737 2801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:22.528033 2801 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:40939
I20260928 23:46:22.528695 2868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:40939 every 8 connection(s)
I20260928 23:46:22.529131 2801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:22.532531 2869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:22.532618 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2801
I20260928 23:46:22.532747 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:46:22.533085 21953 external_mini_cluster.cc:1428] Setting key 3480a7a5c3d47446387b04a79005c1a9
I20260928 23:46:22.538561 2869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.112.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.125@KRBTEST.COM" created.
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:22.557194 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.542482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.21.112.126:40939)
0928 23:46:22.542768 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:22.542779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:22.542815 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:46:22.543708 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:22.543713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:22.543922 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:22.544140 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:22.544923 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.546103 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.547657 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.548437 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.548647 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.551507 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:22.551517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:22.551537 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:22.551573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:22.553806 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.554294 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.554304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.554308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.554483 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.554870 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.554877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.554881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.555464 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:22.555616 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:22.556076 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 23:46:22.556387 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":170,"spinlock_wait_cycles":4352,"thread_start_us":99,"threads_started":1}
I20260928 23:46:22.557200 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.541686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40939 (local address 127.0.0.1:36196)
0928 23:46:22.542273 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.542343 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:22.543406 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:22.544157 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:22.544161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:22.544486 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:22.544756 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.544769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.546249 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.546255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.547519 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.547532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.548603 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.548606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.548686 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.549389 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:22.549406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:22.551333 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:22.553928 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.553936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.553950 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.554171 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.554620 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.554623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.554625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.554751 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.555606 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:22.555612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:22.555750 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 23:46:22.555912 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:46:22.556261 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":50,"spinlock_wait_cycles":3200,"thread_start_us":123,"threads_started":1}
I20260928 23:46:22.560271 2869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:46:22.561169 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.560807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36908)
0928 23:46:22.560986 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.561094 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 23:46:22.561419 2820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
I20260928 23:46:22.562486 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.562172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36910)
0928 23:46:22.562330 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.562405 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 23:46:22.562644 2869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:42793: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab.
Entry for principal kudu/127.21.112.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab.
I20260928 23:46:22.566087 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:42793
--webserver_interface=127.21.112.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab
--principal=kudu/127.21.112.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:40939,127.21.112.125:42793,127.21.112.124:42525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:22.611941 2869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } attempt: 1
I20260928 23:46:22.612918 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.612595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36922)
0928 23:46:22.612768 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.612835 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 23:46:22.613735 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.613481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36924)
0928 23:46:22.613613 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.613672 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:46:22.613869 2869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:42793: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
W20260928 23:46:22.678795 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:22.679100 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:22.679190 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 23:46:22.682049 2869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } attempt: 2
I20260928 23:46:22.682942 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.682635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36928)
0928 23:46:22.682790 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.682863 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 23:46:22.683558 2876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:22.683635 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:22.683686 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:22.683741 2876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260928 23:46:22.683717 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.683459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36932)
0928 23:46:22.683568 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.683625 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:46:22.683780 2876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:46:22.683876 2869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:42793: Network error: Client connection negotiation failed: client connection to 127.21.112.125:42793: connect: Connection refused (error 111)
I20260928 23:46:22.689147 2876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:40939,127.21.112.125:42793,127.21.112.124:42525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab
--principal=kudu/127.21.112.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.112.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:22.690388 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:22.691474 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:22.700850 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:22.700947 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:22.700846 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:22.700969 2876 server_base.cc:1034] running on GCE node
I20260928 23:46:22.701820 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:22.703027 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:22.704245 2876 hybrid_clock.cc:648] HybridClock initialized: now 1790639182704221 us; error 37 us; skew 500 ppm
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:22.707512 2876 init.cc:377] Logged in from keytab as kudu/127.21.112.125@KRBTEST.COM (short username kudu)
I20260928 23:46:22.708472 2876 webserver.cc:460] Webserver started at http://127.21.112.125:43255/ using document root <none> and password file <none>
I20260928 23:46:22.709069 2876 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:22.709160 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:22.709398 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:22.711591 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data/instance:
uuid: "8dc20a82f6f7448e96e3b46c2e9210d8"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "e50253e1f5374c8e424d39cba74ff441"
server_key_iv: "d63244b6ff9a609e6b661a42fde8e7dc"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.712172 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal/instance:
uuid: "8dc20a82f6f7448e96e3b46c2e9210d8"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "e50253e1f5374c8e424d39cba74ff441"
server_key_iv: "d63244b6ff9a609e6b661a42fde8e7dc"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.716039 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:46:22.718549 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:22.719870 2876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:46:22.720099 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal
uuid: "8dc20a82f6f7448e96e3b46c2e9210d8"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "e50253e1f5374c8e424d39cba74ff441"
server_key_iv: "d63244b6ff9a609e6b661a42fde8e7dc"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.720227 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:22.739467 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:22.743633 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:22.743896 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:22.752249 2876 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.125:42793
I20260928 23:46:22.752599 2943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.125:42793 every 8 connection(s)
I20260928 23:46:22.753515 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
I20260928 23:46:22.756578 2944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:22.760632 2944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:46:22.762930 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2876
I20260928 23:46:22.763082 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-1/wal/instance
I20260928 23:46:22.763396 21953 external_mini_cluster.cc:1428] Setting key cf2879cbdf1d66a4686713e18d65de6b
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.124@KRBTEST.COM" created.
I20260928 23:46:22.775032 2869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } attempt: 3
I20260928 23:46:22.776845 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.762839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.21.112.126:40939)
0928 23:46:22.762991 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:22.762995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:22.764361 (+ 1366us) server_negotiation.cc:407] Connection header received
0928 23:46:22.765594 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:22.765600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:22.765667 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:22.765753 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:22.767255 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.767792 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.768970 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.769229 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.769349 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.772096 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:22.772099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:22.772101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:22.772135 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:22.773803 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.774425 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.774429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.774431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.774522 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.774912 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.774917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.774919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.775228 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:22.775339 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:22.776456 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0928 23:46:22.776637 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:46:22.778810 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.763580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40939 (local address 127.0.0.1:36198)
0928 23:46:22.764189 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.764236 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:22.765262 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:22.765990 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:22.766001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:22.766525 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:22.767085 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.767099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.767932 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.767940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.768828 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.768837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.769395 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.769399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.769638 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.770275 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:22.770293 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:22.771886 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:22.773960 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.773970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.773991 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.774304 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.774635 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.774642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.774644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.774804 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.775606 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:22.775614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:22.776093 (+ 479us) client_negotiation.cc:769] Sending connection context
0928 23:46:22.777097 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0928 23:46:22.777600 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":203,"threads_started":1}
I20260928 23:46:22.781147 2944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:46:22.788169 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.775597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36948)
0928 23:46:22.775816 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.775831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:22.775917 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:22.776754 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:22.776757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:22.776946 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:22.777103 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.777111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.779091 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.779095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.779923 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.779937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.780864 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.780871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.781030 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.781708 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:22.781728 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:22.783434 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:22.785958 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.785965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.785971 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.786270 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.786761 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.786764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.786767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.786867 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.787442 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:22.787445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:22.787563 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 23:46:22.787762 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 23:46:22.787939 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 23:46:22.788224 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.775765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.21.112.125:42793)
0928 23:46:22.776227 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:22.776240 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:22.776264 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:46:22.776316 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:22.776324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:22.776586 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:22.777577 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:22.777704 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.778885 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.780093 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.780715 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.780838 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.783608 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:22.783620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:22.783639 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:22.783670 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:22.785854 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.786538 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.786545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.786547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.786664 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.786972 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.786976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.786979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.787319 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:22.787494 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:22.787848 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 23:46:22.788011 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":103,"threads_started":1}
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:46:22.790815 2869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:46:22.791754 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.791426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33836)
0928 23:46:22.791633 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.791684 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 23:46:22.792538 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.792260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33850)
0928 23:46:22.792412 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.792466 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 23:46:22.792778 2869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:42525: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
I20260928 23:46:22.793642 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.781831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36954)
0928 23:46:22.781954 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.781996 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:22.782090 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:22.782546 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:22.782550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:22.782750 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:22.782876 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.782885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.784232 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.784234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.784982 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.784992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.785574 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.785578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.785675 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.786127 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:22.786138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:22.788857 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:22.791898 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.791905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.791908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.792212 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.792491 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.792495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.792497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.792602 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.793124 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:22.793127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:22.793216 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 23:46:22.793300 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:46:22.793457 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 23:46:22.793828 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.781900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.21.112.125:42793)
0928 23:46:22.782272 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:22.782280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:22.782291 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:46:22.782319 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:22.782321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:22.782391 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:22.782501 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:22.783020 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.783858 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.785130 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.785399 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.785575 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.789814 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:22.789822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:22.789825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:22.789849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:22.791785 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.792330 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.792333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.792334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.792416 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.792728 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.792731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.792732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.793008 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:22.793099 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:22.793346 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 23:46:22.793588 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":303,"thread_start_us":125,"threads_started":1}
I20260928 23:46:22.794554 2944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:46:22.795222 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.794994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33862)
0928 23:46:22.795096 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.795172 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:46:22.795413 2894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab.
Entry for principal kudu/127.21.112.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab.
I20260928 23:46:22.796301 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.796084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33866)
0928 23:46:22.796202 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.796250 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 23:46:22.796514 2944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:42525: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
I20260928 23:46:22.796832 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:42525
--webserver_interface=127.21.112.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab
--principal=kudu/127.21.112.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:40939,127.21.112.125:42793,127.21.112.124:42525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:22.824062 2869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } attempt: 1
I20260928 23:46:22.825044 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.824717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33882)
0928 23:46:22.824898 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.824963 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 23:46:22.825917 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.825676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33896)
0928 23:46:22.825795 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.825857 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:46:22.826119 2869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:42525: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
I20260928 23:46:22.845788 2944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } attempt: 1
I20260928 23:46:22.846720 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.846371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33908)
0928 23:46:22.846551 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.846626 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 23:46:22.847540 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.847249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33910)
0928 23:46:22.847383 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.847458 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 23:46:22.847678 2944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:42525: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
I20260928 23:46:22.901309 2869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } attempt: 2
I20260928 23:46:22.902318 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.901973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33924)
0928 23:46:22.902174 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.902240 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 23:46:22.903182 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.902903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33938)
0928 23:46:22.903040 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.903093 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 23:46:22.903357 2869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:42525: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
W20260928 23:46:22.910620 2952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:22.910876 2952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:22.910993 2952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:22.915104 2952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:22.915167 2952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:22.915187 2952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:22.915205 2952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:22.915241 2952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:22.915828 2944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } attempt: 2
I20260928 23:46:22.916662 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.916337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33942)
0928 23:46:22.916501 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.916571 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 23:46:22.917363 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.917151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33948)
0928 23:46:22.917256 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.917307 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:46:22.917484 2944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:42525: Network error: Client connection negotiation failed: client connection to 127.21.112.124:42525: connect: Connection refused (error 111)
I20260928 23:46:22.921247 2952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:40939,127.21.112.125:42793,127.21.112.124:42525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab
--principal=kudu/127.21.112.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:42525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.112.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:22.922413 2952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:22.923457 2952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:22.932057 2957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:22.932057 2958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:22.932096 2960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:22.932313 2952 server_base.cc:1034] running on GCE node
I20260928 23:46:22.933097 2952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:22.934381 2952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:22.935599 2952 hybrid_clock.cc:648] HybridClock initialized: now 1790639182935568 us; error 47 us; skew 500 ppm
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:22.938696 2952 init.cc:377] Logged in from keytab as kudu/127.21.112.124@KRBTEST.COM (short username kudu)
I20260928 23:46:22.939702 2952 webserver.cc:460] Webserver started at http://127.21.112.124:35011/ using document root <none> and password file <none>
I20260928 23:46:22.940290 2952 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:22.940383 2952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:22.940623 2952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:22.942821 2952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data/instance:
uuid: "7a78963a28154e6b9ba709e4fa9f8ebc"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "c1d97adfa34bd998ffa79784b7c90265"
server_key_iv: "33a53cbc25bd27be29d336deb91f149b"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.943397 2952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal/instance:
uuid: "7a78963a28154e6b9ba709e4fa9f8ebc"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "c1d97adfa34bd998ffa79784b7c90265"
server_key_iv: "33a53cbc25bd27be29d336deb91f149b"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.947284 2952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:46:22.949891 2967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:22.951505 2952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:46:22.951665 2952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal
uuid: "7a78963a28154e6b9ba709e4fa9f8ebc"
format_stamp: "Formatted at 2026-09-28 23:46:22 on dist-test-slave-c3s8"
server_key: "c1d97adfa34bd998ffa79784b7c90265"
server_key_iv: "33a53cbc25bd27be29d336deb91f149b"
server_key_version: "encryptionkey@0"
I20260928 23:46:22.951807 2952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:22.966398 2952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:22.969645 2952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:22.969854 2952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:22.976938 2952 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.124:42525
I20260928 23:46:22.977281 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.124:42525 every 8 connection(s)
I20260928 23:46:22.978091 2952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
I20260928 23:46:22.980898 3020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:22.983402 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 2952
I20260928 23:46:22.983547 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790639014951155-21953-0/minicluster-data/master-2/wal/instance
I20260928 23:46:22.983856 21953 external_mini_cluster.cc:1428] Setting key ebf350f58961f3b2d58dbdae9de3284f
I20260928 23:46:22.985060 3020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:46:22.997412 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.987425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.21.112.126:40939)
0928 23:46:22.987567 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:22.987572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:22.987924 (+ 352us) server_negotiation.cc:407] Connection header received
0928 23:46:22.988133 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:22.988136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:22.988182 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:22.988273 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:22.989039 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.989523 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.990648 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.990858 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.990918 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.993390 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:22.993393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:22.993395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:22.993418 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:22.995244 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.995835 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.995839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.995840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.995931 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.996271 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.996275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.996276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.996481 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:22.996591 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:22.997114 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 23:46:22.997237 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 23:46:22.998374 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.987234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:40939 (local address 127.0.0.1:36200)
0928 23:46:22.987798 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:22.987848 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:22.988013 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:22.988331 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:22.988336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:22.988714 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:22.988893 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.988902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.989699 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.989702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.990427 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:22.990470 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.991081 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.991084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:22.991169 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.991717 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:22.991733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:22.993187 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:22.995376 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.995383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.995391 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.995733 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.996018 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:22.996023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:22.996025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:22.996167 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:22.996590 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:22.996596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:22.996866 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 23:46:22.997048 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 23:46:22.997364 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":161,"threads_started":1}
I20260928 23:46:22.998380 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:22.985311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.21.112.124:42525)
0928 23:46:22.985832 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:22.985852 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:22.985895 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:46:22.986765 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:22.986789 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:22.987202 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:22.987595 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:22.988418 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.989410 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.990257 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:22.990741 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:22.990911 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:22.993843 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:22.993848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:22.993858 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:22.993912 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:22.995846 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.996271 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.996277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.996279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.996365 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:22.996663 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:22.996667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:22.996669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:22.996916 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:22.997135 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:22.997350 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 23:46:22.997492 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":180,"threads_started":1}
I20260928 23:46:22.999568 3020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:46:23.002564 2869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } attempt: 3
Sep 28 23:46:22 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
Sep 28 23:46:23 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
Sep 28 23:46:23 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:46:23.008654 2944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } attempt: 3
I20260928 23:46:23.009459 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.000103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:42793 (local address 127.0.0.1:36960)
0928 23:46:23.000228 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:23.000243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:23.000321 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:23.000766 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:23.000771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:23.000952 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:23.001044 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.001049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.001899 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.001906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.002459 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.002468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.003191 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.003196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.003297 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.003841 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:23.003851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:23.005491 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:23.007785 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.007791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.007793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.008010 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.008335 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.008339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.008341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.008436 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.008919 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:23.008924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:23.009018 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 23:46:23.009138 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 23:46:23.009274 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:46:23.009944 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.000226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.21.112.125:42793)
0928 23:46:23.000355 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.000361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.000376 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:46:23.000418 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.000421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.000465 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.000744 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.001171 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.001793 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.002671 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.003028 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.003132 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.005647 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.005651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.005654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.005697 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.007672 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.008140 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.008143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.008145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.008240 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.008516 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.008521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.008523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.008821 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.008916 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.009406 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.009835 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:46:23.010674 3020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:46:23.011271 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.000857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.21.112.126:40939)
0928 23:46:23.000987 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.000991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.001088 (+ 97us) server_negotiation.cc:407] Connection header received
0928 23:46:23.001222 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.001225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.001263 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.001359 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.002041 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.002576 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.003591 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.003891 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.004020 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.006811 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.006817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.006824 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.006846 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.008704 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.009591 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.009594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.009596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.009687 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.010019 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.010025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.010028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.010265 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.010471 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.010900 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.011034 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":52}
I20260928 23:46:23.014075 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.002949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33970)
0928 23:46:23.003042 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:23.003058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:23.003134 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:23.003754 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:23.003757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:23.003936 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:23.004052 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.004058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.004910 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.004916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.005565 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.005579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.006240 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.006243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.006333 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.006768 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:23.006783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:23.008747 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:23.012337 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.012340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.012342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.012590 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.012936 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.012941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.012942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.013036 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.013486 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:23.013489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:23.013621 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 23:46:23.013703 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:46:23.013842 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260928 23:46:23.014729 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.003110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.21.112.124:42525)
0928 23:46:23.003540 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.003545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.003557 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:46:23.003589 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.003595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.003650 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.003728 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.004153 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.004800 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.005710 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.006088 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.006554 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.009207 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.009213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.009219 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.009248 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.012217 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.012745 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.012748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.012749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.012830 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.013164 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.013167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.013168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.013379 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.014363 (+ 984us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.014443 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.014570 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
Sep 28 23:46:23 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
Sep 28 23:46:23 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
Sep 28 23:46:23 dist-test-slave-c3s8 krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:46:23.018878 2869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c: Bootstrap starting.
I20260928 23:46:23.021660 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.011151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33996)
0928 23:46:23.011423 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:23.011440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:23.011505 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:23.011900 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:23.011905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:23.012160 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:23.012346 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.012352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.013382 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.013385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.014178 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.014188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.014915 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.014918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.015031 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.015494 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:23.015506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:23.017400 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:23.019659 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.019662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.019666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.020267 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.020572 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.020575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.020577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.020659 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.021114 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:23.021118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:23.021208 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 23:46:23.021295 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:46:23.021443 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":20,"spinlock_wait_cycles":242432}
I20260928 23:46:23.022321 2869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:23.022346 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.011178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.21.112.124:42525)
0928 23:46:23.011701 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.011710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.011722 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:46:23.011753 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.011755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.011790 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.012025 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.012527 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.013281 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.014294 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.014793 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.015228 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.017571 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.017575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.017578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.017605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.019542 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.020372 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.020379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.020383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.020474 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.020760 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.020766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.020769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.020992 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.021975 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.022048 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.022171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":460,"thread_start_us":127,"threads_started":1}
I20260928 23:46:23.022549 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.009057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:42525 (local address 127.0.0.1:33980)
0928 23:46:23.009197 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:23.009213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:23.009283 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:23.011548 (+ 2265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:23.011554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:23.011751 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:23.011868 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.011878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.013022 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.013026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.013882 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:23.013892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.015158 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.015164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:23.015256 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.015635 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:23.015688 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:23.017520 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:23.020787 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.020792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.020796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.021186 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.021467 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:23.021470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:23.021472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:23.021609 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:23.022067 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:23.022070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:23.022170 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 23:46:23.022252 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:46:23.022389 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 23:46:23.023134 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.009091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.21.112.124:42525)
0928 23:46:23.010863 (+ 1772us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.010876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.010888 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:46:23.010919 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.010922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.011292 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.011974 (+ 682us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.012080 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.012905 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.014038 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.015031 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.015424 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.017693 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.017697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.017700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.017725 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.020673 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.021292 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.021295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.021296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.021366 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.021728 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.021731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.021732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.021954 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.022810 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.022882 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.022989 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":314,"server-negotiator.queue_time_us":1704,"thread_start_us":116,"threads_started":1}
I20260928 23:46:23.023330 2869 log.cc:824] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:23.024159 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.012993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.21.112.125:42793)
0928 23:46:23.013118 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.013122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.013311 (+ 189us) server_negotiation.cc:407] Connection header received
0928 23:46:23.013480 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.013483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.013524 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.013602 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.014478 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.015153 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.016204 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.016545 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.016621 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.019483 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.019487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.019490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.019513 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.021171 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.021672 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.021675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.021677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.021757 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.022829 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.022833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.022834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.023057 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.023268 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.023886 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.023994 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:46:23.025411 3020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: Bootstrap starting.
I20260928 23:46:23.025957 2944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: Bootstrap starting.
I20260928 23:46:23.027122 2869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c: No bootstrap required, opened a new log
I20260928 23:46:23.027807 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:46:23.028146 3020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:23.028208 2944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:23.029036 2944 log.cc:824] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:23.029036 3020 log.cc:824] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:23.032112 2869 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.032471 2869 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:23.032572 2869 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec14c124ad9c484085fa7f73d266fd0c, State: Initialized, Role: FOLLOWER
I20260928 23:46:23.033377 2869 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.034996 2869 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:23.039866 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.030020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.21.112.126:40939)
0928 23:46:23.030233 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.030238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.030786 (+ 548us) server_negotiation.cc:407] Connection header received
0928 23:46:23.030953 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.030956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.031001 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.031050 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.032023 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.032775 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.033962 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.034261 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.034334 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.036260 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.036265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.036269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.036298 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.037997 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.038490 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.038493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.038495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.038570 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.038880 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.038885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.038887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.039104 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.039224 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.039605 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.039722 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:46:23.040336 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.030080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.21.112.124:42525)
0928 23:46:23.030507 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.030512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.030700 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:46:23.030897 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.030900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.030948 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.031001 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.032209 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.032979 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.034381 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.034785 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.034860 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.036664 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.036669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.036674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.036703 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.038466 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.038966 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.038972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.038977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.039088 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.039417 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.039420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.039422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.039618 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.039714 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.040083 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.040200 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260928 23:46:23.040627 2944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: No bootstrap required, opened a new log
I20260928 23:46:23.040704 3020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: No bootstrap required, opened a new log
W20260928 23:46:23.044296 3048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:46:23.044437 3048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:46:23.045207 3020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.045290 2944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.045437 3020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:23.045507 3020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a78963a28154e6b9ba709e4fa9f8ebc, State: Initialized, Role: FOLLOWER
I20260928 23:46:23.045989 3049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.046201 3049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:46:23.046175 3020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.046394 3049 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:23.046578 3049 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.047539 3020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:23.048074 2944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:23.048208 2944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc20a82f6f7448e96e3b46c2e9210d8, State: Initialized, Role: FOLLOWER
I20260928 23:46:23.048892 2944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.051275 2918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec14c124ad9c484085fa7f73d266fd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" is_pre_election: true
I20260928 23:46:23.048038 2801 master_runner.cc:186] Error getting master registration for 127.21.112.125:42793: OK, instance_id { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" instance_seqno: 1790639182749983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:46:23.051561 2918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec14c124ad9c484085fa7f73d266fd0c in term 0.
I20260928 23:46:23.051630 2801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:46:23.051964 3049 leader_election.cc:288] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc20a82f6f7448e96e3b46c2e9210d8 (127.21.112.125:42793), 7a78963a28154e6b9ba709e4fa9f8ebc (127.21.112.124:42525)
I20260928 23:46:23.052680 2994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec14c124ad9c484085fa7f73d266fd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" is_pre_election: true
I20260928 23:46:23.052974 2994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec14c124ad9c484085fa7f73d266fd0c in term 0.
I20260928 23:46:23.053380 2819 leader_election.cc:302] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a78963a28154e6b9ba709e4fa9f8ebc, 8dc20a82f6f7448e96e3b46c2e9210d8, ec14c124ad9c484085fa7f73d266fd0c; no voters:
I20260928 23:46:23.051209 2944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:23.053892 3037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:46:23.053989 3037 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:46:23.054036 3037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:23.055414 3037 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.056118 3037 leader_election.cc:288] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [CANDIDATE]: Term 1 election: Requested vote from peers 8dc20a82f6f7448e96e3b46c2e9210d8 (127.21.112.125:42793), 7a78963a28154e6b9ba709e4fa9f8ebc (127.21.112.124:42525)
I20260928 23:46:23.056197 2994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec14c124ad9c484085fa7f73d266fd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc"
I20260928 23:46:23.056306 2994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:23.053874 2952 master_runner.cc:186] Error getting master registration for 127.21.112.125:42793: OK, instance_id { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" instance_seqno: 1790639182749983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:46:23.057039 2952 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:46:23.057602 2994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec14c124ad9c484085fa7f73d266fd0c in term 1.
I20260928 23:46:23.057719 2918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec14c124ad9c484085fa7f73d266fd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8"
I20260928 23:46:23.057859 2918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:23.057930 2819 leader_election.cc:302] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a78963a28154e6b9ba709e4fa9f8ebc, ec14c124ad9c484085fa7f73d266fd0c; no voters:
I20260928 23:46:23.058095 3037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:23.058238 3037 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [term 1 LEADER]: Becoming Leader. State: Replica: ec14c124ad9c484085fa7f73d266fd0c, State: Running, Role: LEADER
I20260928 23:46:23.058554 3037 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } }
I20260928 23:46:23.059144 2918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec14c124ad9c484085fa7f73d266fd0c in term 1.
I20260928 23:46:23.060372 3037 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec14c124ad9c484085fa7f73d266fd0c. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.060489 3037 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:23.061414 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:23.030095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.21.112.125:42793)
0928 23:46:23.030209 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:23.030213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:23.030761 (+ 548us) server_negotiation.cc:407] Connection header received
0928 23:46:23.030915 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:23.030918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:23.030961 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:23.031010 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:23.032209 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.032982 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.034373 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:23.034731 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:23.035139 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:23.038062 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:23.038067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:23.038072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:23.038099 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:23.047708 (+ 9609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.059778 (+ 12070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.059783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.059785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.059889 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:23.060261 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:23.060267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:23.060269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:23.060519 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:23.060687 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:23.061140 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 23:46:23.061268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:46:23.061969 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.062098 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:46:23.062007 3062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.062233 3062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:46:23.065441 3063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:46:23.065441 3074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:46:23.065516 3063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 23:46:23.065516 3074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:46:23.068058 3076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:23.069036 3076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:23.077364 2994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [term 1 FOLLOWER]: Refusing update from remote peer ec14c124ad9c484085fa7f73d266fd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:23.077472 2918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [term 1 FOLLOWER]: Refusing update from remote peer ec14c124ad9c484085fa7f73d266fd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:23.078079 3049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:23.078269 3037 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:23.084350 3062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec14c124ad9c484085fa7f73d266fd0c. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.084473 3062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:46:23.087736 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec14c124ad9c484085fa7f73d266fd0c. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.087869 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:46:23.094828 3037 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.094967 3037 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:23.095079 3079 mvcc.cc:205] Tried to move back new op lower bound from 7334458093876457472 to 7334458093811015680. Current Snapshot: MvccSnapshot[applied={T|T < 7334458093876457472}]
I20260928 23:46:23.096452 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.096565 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:46:23.098920 3082 mvcc.cc:205] Tried to move back new op lower bound from 7334458093876457472 to 7334458093811015680. Current Snapshot: MvccSnapshot[applied={T|T < 7334458093876457472}]
I20260928 23:46:23.100994 3049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.101131 3049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:23.101951 3062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec14c124ad9c484085fa7f73d266fd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec14c124ad9c484085fa7f73d266fd0c" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 40939 } } peers { permanent_uuid: "8dc20a82f6f7448e96e3b46c2e9210d8" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 42793 } } peers { permanent_uuid: "7a78963a28154e6b9ba709e4fa9f8ebc" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 42525 } } }
I20260928 23:46:23.102079 3062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:46:23.103479 3080 mvcc.cc:205] Tried to move back new op lower bound from 7334458093876457472 to 7334458093811015680. Current Snapshot: MvccSnapshot[applied={T|T < 7334458093876457472}]
I20260928 23:46:23.103935 3076 catalog_manager.cc:1347] Generated new cluster ID: 1cf5844d276e4b50b320ce2003f78b44
I20260928 23:46:23.104001 3076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:23.125691 3076 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:46:23.126991 3076 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:23.141562 3076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec14c124ad9c484085fa7f73d266fd0c: Generated new TSK 0
I20260928 23:46:23.142530 3076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:46:24.067179 3063 catalog_manager.cc:1259] Loaded cluster ID: 1cf5844d276e4b50b320ce2003f78b44
I20260928 23:46:24.067179 3074 catalog_manager.cc:1259] Loaded cluster ID: 1cf5844d276e4b50b320ce2003f78b44
I20260928 23:46:24.067286 3063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: loading cluster ID for follower catalog manager: success
I20260928 23:46:24.067286 3074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: loading cluster ID for follower catalog manager: success
I20260928 23:46:24.069711 3063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: acquiring CA information for follower catalog manager: success
I20260928 23:46:24.069711 3074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: acquiring CA information for follower catalog manager: success
I20260928 23:46:24.071313 3074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8dc20a82f6f7448e96e3b46c2e9210d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:46:24.071313 3063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a78963a28154e6b9ba709e4fa9f8ebc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:46:24.212330 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2801
I20260928 23:46:24.218850 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2876
I20260928 23:46:24.224707 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 2952
2026-09-28T23:46:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3069 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:46:24 dist-test-slave-c3s8 krb5kdc[3094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:46:24 dist-test-slave-c3s8 krb5kdc[3094](info): set up 2 sockets
Sep 28 23:46:24 dist-test-slave-c3s8 krb5kdc[3094](info): commencing operation
krb5kdc: starting...
W20260928 23:46:25.290292 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:46:25 dist-test-slave-c3s8 krb5kdc[3094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:46:25.426218 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41605
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:25.539705 3107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:25.539994 3107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:25.540035 3107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:25.544286 3107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:25.544353 3107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:25.544373 3107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:25.544391 3107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:25.544406 3107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:25.549870 3107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:25.550951 3107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:25.552075 3107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:25.561096 3113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:25.561228 3115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:25.561156 3107 server_base.cc:1034] running on GCE node
W20260928 23:46:25.561105 3112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:25.561996 3107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:25.563298 3107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:25.564486 3107 hybrid_clock.cc:648] HybridClock initialized: now 1790639185564450 us; error 47 us; skew 500 ppm
Sep 28 23:46:25 dist-test-slave-c3s8 krb5kdc[3094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639185, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:25.567584 3107 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:25.568603 3107 webserver.cc:460] Webserver started at http://127.21.112.126:34439/ using document root <none> and password file <none>
I20260928 23:46:25.569206 3107 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:25.569300 3107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:25.569590 3107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:25.572146 3107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "1f247350c3884a629b349cd5c2892ad2"
format_stamp: "Formatted at 2026-09-28 23:46:25 on dist-test-slave-c3s8"
server_key: "9934dd875083bc80a1c29c5bf4d0a3ab"
server_key_iv: "b17f036c92c5127f15a1097698c8d782"
server_key_version: "encryptionkey@0"
I20260928 23:46:25.572824 3107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "1f247350c3884a629b349cd5c2892ad2"
format_stamp: "Formatted at 2026-09-28 23:46:25 on dist-test-slave-c3s8"
server_key: "9934dd875083bc80a1c29c5bf4d0a3ab"
server_key_iv: "b17f036c92c5127f15a1097698c8d782"
server_key_version: "encryptionkey@0"
I20260928 23:46:25.576735 3107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:46:25.579493 3122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:25.580919 3107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:46:25.581090 3107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "1f247350c3884a629b349cd5c2892ad2"
format_stamp: "Formatted at 2026-09-28 23:46:25 on dist-test-slave-c3s8"
server_key: "9934dd875083bc80a1c29c5bf4d0a3ab"
server_key_iv: "b17f036c92c5127f15a1097698c8d782"
server_key_version: "encryptionkey@0"
I20260928 23:46:25.581240 3107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:25.606153 3107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:25.609495 3107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:25.609745 3107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:25.617507 3107 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:41605
I20260928 23:46:25.617522 3174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:41605 every 8 connection(s)
I20260928 23:46:25.618630 3107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:25.621973 3175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:25.623214 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3107
I20260928 23:46:25.623366 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:46:25.623682 21953 external_mini_cluster.cc:1428] Setting key b31ef7ad7aa996aa8be8b671defa8981
I20260928 23:46:25.628584 3175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2: Bootstrap starting.
I20260928 23:46:25.631219 3175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:25.632036 3175 log.cc:824] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:46:25 dist-test-slave-c3s8 krb5kdc[3094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:25.634651 3175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2: No bootstrap required, opened a new log
I20260928 23:46:25.638510 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:25.625158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.21.112.126:41605)
0928 23:46:25.625692 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:25.625708 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:25.625744 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:46:25.626413 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:25.626431 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:25.626764 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:25.627097 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:25.627889 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:25.628914 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:25.629879 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:25.630357 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:25.630529 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:25.633431 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:25.633435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:25.633448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:25.633473 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:25.635386 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:25.635931 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:25.635936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:25.635941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:25.636068 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:25.636378 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:25.636380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:25.636381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:25.636807 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:25.636954 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:25.637354 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 23:46:25.637722 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260928 23:46:25.639164 3175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f247350c3884a629b349cd5c2892ad2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41605 } }
I20260928 23:46:25.639400 3175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:25.639489 3175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f247350c3884a629b349cd5c2892ad2, State: Initialized, Role: FOLLOWER
I20260928 23:46:25.639974 3175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f247350c3884a629b349cd5c2892ad2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41605 } }
I20260928 23:46:25.640110 3175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:25.640204 3175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:25.640318 3175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:25.641497 3175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f247350c3884a629b349cd5c2892ad2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41605 } }
I20260928 23:46:25.641582 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:46:25.641923 3175 leader_election.cc:302] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f247350c3884a629b349cd5c2892ad2; no voters:
I20260928 23:46:25.642316 3175 leader_election.cc:288] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:46:25.642441 3180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:25.642671 3180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 1f247350c3884a629b349cd5c2892ad2, State: Running, Role: LEADER
I20260928 23:46:25.643055 3180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f247350c3884a629b349cd5c2892ad2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41605 } }
I20260928 23:46:25.643814 3175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:25.644910 3182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f247350c3884a629b349cd5c2892ad2. Latest consensus state: current_term: 1 leader_uuid: "1f247350c3884a629b349cd5c2892ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f247350c3884a629b349cd5c2892ad2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41605 } } }
I20260928 23:46:25.645104 3182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:25.645417 3181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f247350c3884a629b349cd5c2892ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f247350c3884a629b349cd5c2892ad2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41605 } } }
I20260928 23:46:25.645514 3189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:25.645641 3181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:25.648429 3189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:25.654598 3189 catalog_manager.cc:1347] Generated new cluster ID: d85f80a466b9402b82d45ccf22e4995a
I20260928 23:46:25.654688 3189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:25.669622 3189 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:46:25.671052 3189 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:25.686605 3189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f247350c3884a629b349cd5c2892ad2: Generated new TSK 0
I20260928 23:46:25.687563 3189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:46:25.697984 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3107
2026-09-28T23:46:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1471 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:46:25 dist-test-slave-c3s8 krb5kdc[3202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:46:25 dist-test-slave-c3s8 krb5kdc[3202](info): set up 2 sockets
Sep 28 23:46:25 dist-test-slave-c3s8 krb5kdc[3202](info): commencing operation
krb5kdc: starting...
W20260928 23:46:26.818539 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:46:26 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:46:26.963258 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43455
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:27.075836 3215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:27.076119 3215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:27.076161 3215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:27.080225 3215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:27.080295 3215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:27.080315 3215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:27.080332 3215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:27.080348 3215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:27.085916 3215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:27.087124 3215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:27.088222 3215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:27.097720 3221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:27.097890 3215 server_base.cc:1034] running on GCE node
W20260928 23:46:27.097756 3223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:27.097697 3220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:27.098624 3215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:27.099788 3215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:27.101017 3215 hybrid_clock.cc:648] HybridClock initialized: now 1790639187100993 us; error 36 us; skew 500 ppm
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:27.103952 3215 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:27.104866 3215 webserver.cc:460] Webserver started at http://127.21.112.126:45485/ using document root <none> and password file <none>
I20260928 23:46:27.105458 3215 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:27.105571 3215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:27.105844 3215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:27.107954 3215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "71db0fe9dc9a47c8953a95ebd297e9b2"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "8727551b1ef0dcd250d4a13ebee2032d"
server_key_iv: "22fefc93355e18e59b3147c4d10bf1e3"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.108525 3215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "71db0fe9dc9a47c8953a95ebd297e9b2"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "8727551b1ef0dcd250d4a13ebee2032d"
server_key_iv: "22fefc93355e18e59b3147c4d10bf1e3"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.112494 3215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:46:27.115288 3230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.116690 3215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:46:27.116832 3215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "71db0fe9dc9a47c8953a95ebd297e9b2"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "8727551b1ef0dcd250d4a13ebee2032d"
server_key_iv: "22fefc93355e18e59b3147c4d10bf1e3"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.116952 3215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:27.135893 3215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:27.138960 3215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:27.139192 3215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:27.146566 3215 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:43455
I20260928 23:46:27.146884 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:43455 every 8 connection(s)
I20260928 23:46:27.147590 3215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:27.150171 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3215
I20260928 23:46:27.150326 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:46:27.150579 3283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:27.150604 21953 external_mini_cluster.cc:1428] Setting key ad0d7f3134daf6f87afe8b1494c82907
I20260928 23:46:27.157188 3283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap starting.
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:27.159891 3283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:27.160662 3283 log.cc:824] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:27.163093 3283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: No bootstrap required, opened a new log
I20260928 23:46:27.165412 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.151990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.21.112.126:43455)
0928 23:46:27.152418 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.152433 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.152467 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:46:27.153403 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.153422 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.153817 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.154172 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:27.155010 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.156005 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.156896 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.157444 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.157636 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.160383 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:27.160388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:27.160402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:27.160430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:27.162255 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.162702 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.162709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.162714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.162821 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.163137 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.163142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.163145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.163682 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:27.163879 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.164258 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.164584 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260928 23:46:27.168550 3283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:27.168794 3283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:27.168879 3283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db0fe9dc9a47c8953a95ebd297e9b2, State: Initialized, Role: FOLLOWER
I20260928 23:46:27.169385 3283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:27.169575 3283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:27.169655 3283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:27.169770 3283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:27.170817 3283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:27.171201 3283 leader_election.cc:302] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71db0fe9dc9a47c8953a95ebd297e9b2; no voters:
I20260928 23:46:27.171546 3283 leader_election.cc:288] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:46:27.171676 3288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:27.171940 3288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 LEADER]: Becoming Leader. State: Replica: 71db0fe9dc9a47c8953a95ebd297e9b2, State: Running, Role: LEADER
I20260928 23:46:27.172269 3288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:27.172972 3283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:27.174237 3289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } } }
I20260928 23:46:27.174233 3290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71db0fe9dc9a47c8953a95ebd297e9b2. Latest consensus state: current_term: 1 leader_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } } }
I20260928 23:46:27.174415 3289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:27.174480 3290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:27.174927 3297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:27.177609 3297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:27.183525 3297 catalog_manager.cc:1347] Generated new cluster ID: 4e0900f3514f4f69959f319a8521ee3f
I20260928 23:46:27.183602 3297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:27.198612 3297 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:46:27.199996 3297 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:27.216567 3297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Generated new TSK 0
I20260928 23:46:27.217576 3297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:46:27.254388 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43455
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:27.381629 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:27.381906 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:27.381953 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:27.386057 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:27.386108 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:27.386166 3309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:46:27.391428 3309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:27.392421 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:27.393388 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:27.402874 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:27.402909 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:27.403077 3309 server_base.cc:1034] running on GCE node
W20260928 23:46:27.402897 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:27.403568 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:27.404240 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:27.405407 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790639187405390 us; error 28 us; skew 500 ppm
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:27.408255 3309 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:46:27.409170 3309 webserver.cc:460] Webserver started at http://127.21.112.65:41891/ using document root <none> and password file <none>
I20260928 23:46:27.409807 3309 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:27.409871 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:27.410120 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:27.412189 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "d0f121ee55bf4baa92f89b7306dc8c9c"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "3a1252dc8fb35028e11eb48e6f53faca"
server_key_iv: "766c3d075dde350fa49f62e6cdd1b5ff"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.412768 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "d0f121ee55bf4baa92f89b7306dc8c9c"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "3a1252dc8fb35028e11eb48e6f53faca"
server_key_iv: "766c3d075dde350fa49f62e6cdd1b5ff"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.416590 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:46:27.419225 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.420392 3309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 23:46:27.420537 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "d0f121ee55bf4baa92f89b7306dc8c9c"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "3a1252dc8fb35028e11eb48e6f53faca"
server_key_iv: "766c3d075dde350fa49f62e6cdd1b5ff"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.420645 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:27.435709 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:27.438757 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:27.438961 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:27.439575 3309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:27.440402 3309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:27.440447 3309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.440515 3309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:27.440546 3309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.449499 3309 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:45163
I20260928 23:46:27.449527 3437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:45163 every 8 connection(s)
I20260928 23:46:27.450464 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:46:27.451862 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3309
I20260928 23:46:27.452004 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:46:27.452287 21953 external_mini_cluster.cc:1428] Setting key 103878f6a5997a02cb349ea44579d0e0
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:46:27.465776 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.452573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:60531 (local address 127.21.112.126:43455)
0928 23:46:27.452742 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.452747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.453702 (+ 955us) server_negotiation.cc:407] Connection header received
0928 23:46:27.455830 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.455834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.455898 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.456037 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:27.457351 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.457952 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.459171 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.459405 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.459525 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.462235 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:27.462238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:27.462240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:27.462264 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:27.463665 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.464173 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.464178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.464180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.464269 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.464612 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.464623 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.464625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.464802 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:27.464903 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.465445 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.465592 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:46:27.466552 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.452996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:60531)
0928 23:46:27.453512 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:27.453617 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:27.455542 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:27.456191 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:27.456199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:27.456559 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:27.457051 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.457070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.458116 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.458119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.459040 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.459050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.459563 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.459567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.459741 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.460408 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:27.460426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:27.462052 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:27.463793 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:27.463799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:27.463814 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:27.464061 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:27.464366 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:27.464369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:27.464372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:27.464509 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:27.464914 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:27.464921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:27.465200 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 23:46:27.465402 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 23:46:27.465749 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260928 23:46:27.468063 3438 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43455
I20260928 23:46:27.468322 3438 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:27.468906 3438 heartbeater.cc:511] Master 127.21.112.126:43455 requested a full tablet report, sending...
I20260928 23:46:27.470520 3246 ts_manager.cc:194] Registered new tserver with Master: d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163)
I20260928 23:46:27.471673 3246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:60531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:46:27.485116 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43455
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:27.602501 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:27.602769 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:27.602814 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:27.606987 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:27.607059 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:27.607120 3443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:46:27.612861 3443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:27.614104 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:27.615218 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:46:27.626075 3443 server_base.cc:1034] running on GCE node
W20260928 23:46:27.625991 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:27.625981 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:27.625981 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:27.626713 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:27.627409 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:27.628587 3443 hybrid_clock.cc:648] HybridClock initialized: now 1790639187628564 us; error 34 us; skew 500 ppm
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:27.631608 3443 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:46:27.632592 3443 webserver.cc:460] Webserver started at http://127.21.112.66:37769/ using document root <none> and password file <none>
I20260928 23:46:27.633170 3443 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:27.633231 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:27.633400 3443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:27.635629 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "4e0439409b474559acdcdc933b4a8a88"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "5fcd74a4f7c53ce7decc941cda9d737a"
server_key_iv: "63e2bf223e01c84069efb4751ef9a6e1"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.636227 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "4e0439409b474559acdcdc933b4a8a88"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "5fcd74a4f7c53ce7decc941cda9d737a"
server_key_iv: "63e2bf223e01c84069efb4751ef9a6e1"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.640367 3443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 23:46:27.643195 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.644694 3443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:46:27.644804 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "4e0439409b474559acdcdc933b4a8a88"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "5fcd74a4f7c53ce7decc941cda9d737a"
server_key_iv: "63e2bf223e01c84069efb4751ef9a6e1"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.644905 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:27.675791 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:27.679551 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:27.679742 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:27.680326 3443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:27.681219 3443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:27.681270 3443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.681313 3443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:27.681334 3443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.690371 3443 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:38549
I20260928 23:46:27.690388 3571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:38549 every 8 connection(s)
I20260928 23:46:27.691329 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:46:27.692360 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3443
I20260928 23:46:27.692487 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:46:27.692782 21953 external_mini_cluster.cc:1428] Setting key 75e75e8eddef16cdf4e6be36f0b75950
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:27.711416 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.693512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:42711 (local address 127.21.112.126:43455)
0928 23:46:27.693688 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.693693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.694643 (+ 950us) server_negotiation.cc:407] Connection header received
0928 23:46:27.696167 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.696170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.696218 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.696293 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:27.697985 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.698666 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.700662 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.700900 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.701087 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.707177 (+ 6090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:27.707183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:27.707189 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:27.707228 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:27.708830 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.709522 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.709524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.709525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.709665 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.710116 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.710119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.710121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.710311 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:27.710387 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.711181 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.711292 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 23:46:27.712848 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.693945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:42711)
0928 23:46:27.694498 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:27.694543 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:27.695882 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:27.696493 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:27.696505 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:27.697050 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:27.697812 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.697837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.698833 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.698838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.700480 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.700500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.701168 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.701175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.701439 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.702245 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:27.702271 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:27.706954 (+ 4683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:27.708977 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:27.708985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:27.709007 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:27.709409 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:27.709763 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:27.709769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:27.709775 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:27.710009 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:27.710429 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:27.710440 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:27.710850 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 23:46:27.711122 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 23:46:27.711544 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260928 23:46:27.714284 3572 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43455
I20260928 23:46:27.714545 3572 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:27.715085 3572 heartbeater.cc:511] Master 127.21.112.126:43455 requested a full tablet report, sending...
I20260928 23:46:27.716248 3246 ts_manager.cc:194] Registered new tserver with Master: 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
I20260928 23:46:27.716907 3246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:42711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:46:27.725301 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43455
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:27.839815 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:27.840111 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:27.840170 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:27.844362 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:27.844434 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:27.844498 3577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:46:27.850095 3577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:27.851166 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:27.852183 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:27.861860 3582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:27.861876 3585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:27.861856 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:27.862392 3577 server_base.cc:1034] running on GCE node
I20260928 23:46:27.862823 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:27.863523 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:27.864719 3577 hybrid_clock.cc:648] HybridClock initialized: now 1790639187864702 us; error 30 us; skew 500 ppm
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:27.867542 3577 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:46:27.868408 3577 webserver.cc:460] Webserver started at http://127.21.112.67:35183/ using document root <none> and password file <none>
I20260928 23:46:27.868978 3577 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:27.869068 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:27.869303 3577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:27.871491 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "4e75d7000058444ca07fc3bd1540c506"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "b322c0ec90c09137bd921d4f1792b633"
server_key_iv: "c165daf87bb0275d3d9dc11e309c74bf"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.872047 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "4e75d7000058444ca07fc3bd1540c506"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "b322c0ec90c09137bd921d4f1792b633"
server_key_iv: "c165daf87bb0275d3d9dc11e309c74bf"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.876132 3577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:46:27.878669 3592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.879949 3577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:46:27.880082 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "4e75d7000058444ca07fc3bd1540c506"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "b322c0ec90c09137bd921d4f1792b633"
server_key_iv: "c165daf87bb0275d3d9dc11e309c74bf"
server_key_version: "encryptionkey@0"
I20260928 23:46:27.880199 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:27.907948 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:27.911237 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:27.911446 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:27.912125 3577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:27.912982 3577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:27.913072 3577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.913146 3577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:27.913198 3577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:27.922914 3577 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:42177
I20260928 23:46:27.922986 3705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:42177 every 8 connection(s)
I20260928 23:46:27.923872 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:46:27.932235 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3577
I20260928 23:46:27.932395 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:46:27.932663 21953 external_mini_cluster.cc:1428] Setting key 9908eac6baeabb1d97b837653db89c19
Sep 28 23:46:27 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639187, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:27.938562 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.925819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:53871 (local address 127.21.112.126:43455)
0928 23:46:27.925982 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.925987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.926891 (+ 904us) server_negotiation.cc:407] Connection header received
0928 23:46:27.928250 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.928253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.928304 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.928383 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:27.929746 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.930320 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.931454 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.931687 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.931773 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.934851 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:27.934855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:27.934858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:27.934890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:27.936491 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.937010 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.937013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.937014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.937101 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.937412 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.937415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.937416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.937691 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:27.937774 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.938332 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.938441 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:46:27.939529 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.926273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:53871)
0928 23:46:27.926733 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:27.926778 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:27.928034 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:27.928561 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:27.928568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:27.928948 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:27.929515 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.929588 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.930440 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.930442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.931347 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.931356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.931823 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.931825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.931968 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.932719 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:27.932761 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:27.934683 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:27.936606 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:27.936611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:27.936635 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:27.936898 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:27.937196 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:27.937198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:27.937199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:27.937329 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:27.937790 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:27.937797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:27.938110 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 23:46:27.938275 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 23:46:27.938550 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260928 23:46:27.940747 3706 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43455
I20260928 23:46:27.941110 3706 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:27.941718 3706 heartbeater.cc:511] Master 127.21.112.126:43455 requested a full tablet report, sending...
I20260928 23:46:27.942869 3246 ts_manager.cc:194] Registered new tserver with Master: 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177)
I20260928 23:46:27.943454 3246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:53871
I20260928 23:46:27.946873 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:46:27.956399 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.948977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.21.112.126:43455)
0928 23:46:27.949175 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.949180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.949372 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:46:27.949455 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.949457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.949551 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.949636 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:27.950388 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.950915 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.951904 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.952136 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.952247 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.953722 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:27.953725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:27.953727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:27.953750 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:27.954988 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.955376 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.955379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.955380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.955449 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:27.955679 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:27.955682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:27.955683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:27.955882 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:27.955981 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.956190 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.956280 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:46:27.959358 3246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:46:27.962395 3246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:46:27.977360 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.971143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:42177 (local address 127.0.0.1:32866)
0928 23:46:27.972004 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:27.972018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:27.972118 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:27.972929 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:27.972936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:27.972958 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:27.973103 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.973110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.974810 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.974814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.975895 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.975905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.976987 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.976994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.977084 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.977106 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:46:27.977174 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:46:27.977227 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":26,"thread_start_us":96,"threads_started":1}
I20260928 23:46:27.977459 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.971919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.21.112.67:42177)
0928 23:46:27.972392 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.972405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.972428 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:46:27.972508 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.972514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.972748 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.972938 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:27.973237 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.974593 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.976021 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.976868 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.976972 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.977101 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.977206 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.977312 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":186,"threads_started":1}
I20260928 23:46:27.977730 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.971926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.21.112.65:45163)
0928 23:46:27.972426 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.972433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.972458 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:46:27.972580 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.972586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.972792 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.972930 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:27.973454 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.974379 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.975985 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.976546 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.976681 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.976852 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.977426 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.977574 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":94,"threads_started":1}
I20260928 23:46:27.977360 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.971400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:45163 (local address 127.0.0.1:56120)
0928 23:46:27.972323 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:27.972337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:27.972416 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:27.973103 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:27.973105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:27.973133 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:27.973320 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.973329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.974504 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.974510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.975858 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.975868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.976680 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.976683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.976848 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.976899 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:46:27.977031 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 23:46:27.977182 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":165,"thread_start_us":85,"threads_started":1}
I20260928 23:46:27.979032 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.971065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38549 (local address 127.0.0.1:41982)
0928 23:46:27.971809 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:27.971866 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:27.971990 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:27.973168 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:27.973172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:27.973189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:27.973365 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.973372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.974828 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.974831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.977045 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:27.977055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.978626 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.978632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:27.978729 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.978754 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:46:27.978835 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:46:27.978905 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":81,"thread_start_us":136,"threads_started":1}
I20260928 23:46:27.979344 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:27.971183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.21.112.66:38549)
0928 23:46:27.972715 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:27.972725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:27.972745 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:46:27.972816 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:27.972821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:27.973010 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:27.973171 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:27.973517 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.974687 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.977198 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:27.978495 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:27.978604 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:27.978731 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:27.978877 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 23:46:27.978973 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429,"thread_start_us":97,"threads_started":1}
I20260928 23:46:27.980580 3640 tablet_service.cc:1467] Processing CreateTablet for tablet 4a973dbb479b444baf65d4f5cff0831f (DEFAULT_TABLE table=test-table [id=efae263e9a6c47469a0bf9af07623dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:27.980605 3372 tablet_service.cc:1467] Processing CreateTablet for tablet 4a973dbb479b444baf65d4f5cff0831f (DEFAULT_TABLE table=test-table [id=efae263e9a6c47469a0bf9af07623dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:27.981711 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a973dbb479b444baf65d4f5cff0831f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:27.981724 3640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a973dbb479b444baf65d4f5cff0831f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:27.982030 3506 tablet_service.cc:1467] Processing CreateTablet for tablet 4a973dbb479b444baf65d4f5cff0831f (DEFAULT_TABLE table=test-table [id=efae263e9a6c47469a0bf9af07623dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:27.983101 3506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a973dbb479b444baf65d4f5cff0831f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:27.988492 3723 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Bootstrap starting.
I20260928 23:46:27.990381 3724 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap starting.
I20260928 23:46:27.991942 3723 tablet_bootstrap.cc:654] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:27.992266 3724 tablet_bootstrap.cc:654] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:27.992760 3723 log.cc:824] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:27.993273 3724 log.cc:824] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:27.993898 3725 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Bootstrap starting.
I20260928 23:46:27.994968 3723 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: No bootstrap required, opened a new log
I20260928 23:46:27.995208 3723 ts_tablet_manager.cc:1397] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:46:27.995410 3724 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: No bootstrap required, opened a new log
I20260928 23:46:27.995666 3724 ts_tablet_manager.cc:1397] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260928 23:46:27.996622 3725 tablet_bootstrap.cc:654] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:27.997481 3725 log.cc:824] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:27.998586 3723 raft_consensus.cc:348] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:27.998834 3723 raft_consensus.cc:374] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:27.998905 3723 raft_consensus.cc:729] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e75d7000058444ca07fc3bd1540c506, State: Initialized, Role: FOLLOWER
I20260928 23:46:27.999384 3723 consensus_queue.cc:260] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:27.999650 3725 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: No bootstrap required, opened a new log
I20260928 23:46:27.999890 3725 ts_tablet_manager.cc:1397] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:46:28.000272 3706 heartbeater.cc:503] Master 127.21.112.126:43455 was elected leader, sending a full tablet report...
I20260928 23:46:28.000435 3723 ts_tablet_manager.cc:1428] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:46:28.001006 3724 raft_consensus.cc:348] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.001361 3724 raft_consensus.cc:374] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:28.001430 3724 raft_consensus.cc:729] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f121ee55bf4baa92f89b7306dc8c9c, State: Initialized, Role: FOLLOWER
I20260928 23:46:28.001914 3724 consensus_queue.cc:260] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.002545 3438 heartbeater.cc:503] Master 127.21.112.126:43455 was elected leader, sending a full tablet report...
I20260928 23:46:28.002905 3724 ts_tablet_manager.cc:1428] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:46:28.003932 3725 raft_consensus.cc:348] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.004191 3725 raft_consensus.cc:374] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:28.004292 3725 raft_consensus.cc:729] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0439409b474559acdcdc933b4a8a88, State: Initialized, Role: FOLLOWER
I20260928 23:46:28.004763 3725 consensus_queue.cc:260] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.005606 3572 heartbeater.cc:503] Master 127.21.112.126:43455 was elected leader, sending a full tablet report...
I20260928 23:46:28.005824 3725 ts_tablet_manager.cc:1428] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:46:28.152865 3730 raft_consensus.cc:482] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:28.153127 3730 raft_consensus.cc:504] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.154600 3730 leader_election.cc:288] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549), 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177)
I20260928 23:46:28.159662 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.154791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38549 (local address 127.21.112.65:51837)
0928 23:46:28.155241 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:28.155260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:28.155446 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:28.155800 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:28.155806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:28.155835 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:28.156005 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.156012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.157223 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.157226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.158281 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.158292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.159190 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.159193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.159300 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.159327 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:46:28.159403 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:46:28.159479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":51,"thread_start_us":143,"threads_started":1}
I20260928 23:46:28.159701 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.155011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:51837 (local address 127.21.112.66:38549)
0928 23:46:28.155162 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:28.155169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:28.155346 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:46:28.155594 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:28.155598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:28.155655 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:28.155767 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:28.156141 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.157034 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.158428 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.159052 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.159192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.159252 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:28.159448 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:46:28.159541 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:46:28.160420 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.155273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:38651 (local address 127.21.112.67:42177)
0928 23:46:28.155393 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:28.155398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:28.155446 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:46:28.155680 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:28.155685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:28.155773 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:28.155849 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:28.156196 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.157046 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.158702 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.159354 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.159512 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.159561 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:28.160200 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 23:46:28.160294 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:46:28.160580 3526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a973dbb479b444baf65d4f5cff0831f" candidate_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0439409b474559acdcdc933b4a8a88" is_pre_election: true
I20260928 23:46:28.160993 3526 raft_consensus.cc:2454] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f121ee55bf4baa92f89b7306dc8c9c in term 0.
I20260928 23:46:28.161212 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.154874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:42177 (local address 127.21.112.65:38651)
0928 23:46:28.155362 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:28.155377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:28.155479 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:28.155902 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:28.155905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:28.155924 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:28.156049 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.156056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.157180 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.157184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.158550 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.158562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.159627 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.159631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.159744 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.160030 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 23:46:28.160178 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 23:46:28.161035 (+ 857us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260928 23:46:28.161608 3325 leader_election.cc:302] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0439409b474559acdcdc933b4a8a88, d0f121ee55bf4baa92f89b7306dc8c9c; no voters:
I20260928 23:46:28.161988 3730 raft_consensus.cc:2781] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:46:28.161983 3660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a973dbb479b444baf65d4f5cff0831f" candidate_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e75d7000058444ca07fc3bd1540c506" is_pre_election: true
I20260928 23:46:28.162170 3730 raft_consensus.cc:482] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:46:28.162242 3730 raft_consensus.cc:3037] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:28.162323 3660 raft_consensus.cc:2454] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0f121ee55bf4baa92f89b7306dc8c9c in term 0.
I20260928 23:46:28.163494 3730 raft_consensus.cc:504] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.163853 3730 leader_election.cc:288] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 1 election: Requested vote from peers 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549), 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177)
I20260928 23:46:28.164245 3526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a973dbb479b444baf65d4f5cff0831f" candidate_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0439409b474559acdcdc933b4a8a88"
I20260928 23:46:28.164243 3660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a973dbb479b444baf65d4f5cff0831f" candidate_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e75d7000058444ca07fc3bd1540c506"
I20260928 23:46:28.164394 3526 raft_consensus.cc:3037] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:28.164405 3660 raft_consensus.cc:3037] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:28.165426 3660 raft_consensus.cc:2454] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f121ee55bf4baa92f89b7306dc8c9c in term 1.
I20260928 23:46:28.165527 3526 raft_consensus.cc:2454] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f121ee55bf4baa92f89b7306dc8c9c in term 1.
I20260928 23:46:28.165728 3327 leader_election.cc:302] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e75d7000058444ca07fc3bd1540c506, d0f121ee55bf4baa92f89b7306dc8c9c; no voters:
I20260928 23:46:28.165895 3730 raft_consensus.cc:2781] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:28.166127 3730 raft_consensus.cc:686] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 LEADER]: Becoming Leader. State: Replica: d0f121ee55bf4baa92f89b7306dc8c9c, State: Running, Role: LEADER
I20260928 23:46:28.166500 3730 consensus_queue.cc:237] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.170324 3246 catalog_manager.cc:5495] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65). New cstate: current_term: 1 leader_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } health_report { overall_health: HEALTHY } } }
W20260928 23:46:28.174887 3707 tablet.cc:2369] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:46:28.192548 3573 tablet.cc:2369] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:46:28.201872 3439 tablet.cc:2369] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:46:28.222997 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.219253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.21.112.65:45163)
0928 23:46:28.219385 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:28.219389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:28.219402 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:46:28.219480 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:28.219482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:28.219534 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:28.219627 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:46:28.219896 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.220561 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.221451 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.221701 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.221810 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.222151 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:46:28.222612 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:46:28.222751 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:28.222788 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 23:46:28.222849 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:46:28.229048 3526 raft_consensus.cc:1261] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Refusing update from remote peer d0f121ee55bf4baa92f89b7306dc8c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:28.229048 3660 raft_consensus.cc:1261] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Refusing update from remote peer d0f121ee55bf4baa92f89b7306dc8c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:28.229812 3734 consensus_queue.cc:1035] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:28.230106 3730 consensus_queue.cc:1035] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:28.239509 3738 mvcc.cc:205] Tried to move back new op lower bound from 7334458114978922496 to 7334458114733178880. Current Snapshot: MvccSnapshot[applied={T|T < 7334458114978922496}]
I20260928 23:46:28.247169 3246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:28.247426 3246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:28.250542 3246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Sending DeleteTablet for 3 replicas of tablet 4a973dbb479b444baf65d4f5cff0831f
I20260928 23:46:28.251447 3506 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a973dbb479b444baf65d4f5cff0831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:28 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260928 23:46:28.251681 3372 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a973dbb479b444baf65d4f5cff0831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:28 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:56120
I20260928 23:46:28.251693 3640 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a973dbb479b444baf65d4f5cff0831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:28 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:32866
I20260928 23:46:28.252177 3747 tablet_replica.cc:330] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: stopping tablet replica
I20260928 23:46:28.252193 3748 tablet_replica.cc:330] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: stopping tablet replica
I20260928 23:46:28.252528 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3309
I20260928 23:46:28.252537 3747 raft_consensus.cc:2229] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:28.252554 3748 raft_consensus.cc:2229] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:46:28.252925 3747 raft_consensus.cc:2258] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:28.252975 3748 raft_consensus.cc:2258] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:28.253860 3749 tablet_replica.cc:330] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: stopping tablet replica
I20260928 23:46:28.254243 3749 raft_consensus.cc:2229] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:28.254652 3747 ts_tablet_manager.cc:1905] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:28.254727 3749 raft_consensus.cc:2258] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:28.256136 3749 ts_tablet_manager.cc:1905] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:28.258287 3747 ts_tablet_manager.cc:1918] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:28.258404 3747 log.cc:1196] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/4a973dbb479b444baf65d4f5cff0831f
I20260928 23:46:28.258756 3749 ts_tablet_manager.cc:1918] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:28.258807 3747 ts_tablet_manager.cc:1939] T 4a973dbb479b444baf65d4f5cff0831f P 4e0439409b474559acdcdc933b4a8a88: Deleting consensus metadata
I20260928 23:46:28.258877 3749 log.cc:1196] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/4a973dbb479b444baf65d4f5cff0831f
I20260928 23:46:28.259205 3749 ts_tablet_manager.cc:1939] T 4a973dbb479b444baf65d4f5cff0831f P 4e75d7000058444ca07fc3bd1540c506: Deleting consensus metadata
I20260928 23:46:28.260224 3231 catalog_manager.cc:4841] TS 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): tablet 4a973dbb479b444baf65d4f5cff0831f (table test-table [id=efae263e9a6c47469a0bf9af07623dcf]) successfully deleted
I20260928 23:46:28.260478 3233 catalog_manager.cc:4841] TS 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): tablet 4a973dbb479b444baf65d4f5cff0831f (table test-table [id=efae263e9a6c47469a0bf9af07623dcf]) successfully deleted
W20260928 23:46:28.260921 3234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:45163 (error 108)
I20260928 23:46:28.261736 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3443
I20260928 23:46:28.261974 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.261673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:45163 (local address 127.0.0.1:56142)
0928 23:46:28.261807 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:28.261881 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:45163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:46:28.262460 3234 catalog_manager.cc:4584] TS d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a973dbb479b444baf65d4f5cff0831f: Network error: Client connection negotiation failed: client connection to 127.21.112.65:45163: connect: Connection refused (error 111)
I20260928 23:46:28.268114 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3577
I20260928 23:46:28.273506 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3215
I20260928 23:46:28.279953 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43455
--webserver_interface=127.21.112.126
--webserver_port=45485
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:28.393937 3751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:28.394238 3751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:28.394281 3751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:28.398504 3751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:28.398574 3751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:28.398594 3751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:28.398612 3751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:28.398631 3751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:28.404330 3751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=45485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:28.405476 3751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:28.406560 3751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:28.421828 3757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:28.421828 3758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:28.421830 3760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:28.422088 3751 server_base.cc:1034] running on GCE node
I20260928 23:46:28.422784 3751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:28.424103 3751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:28.425287 3751 hybrid_clock.cc:648] HybridClock initialized: now 1790639188425247 us; error 58 us; skew 500 ppm
Sep 28 23:46:28 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:28.428161 3751 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:28.429105 3751 webserver.cc:460] Webserver started at http://127.21.112.126:45485/ using document root <none> and password file <none>
I20260928 23:46:28.429740 3751 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:28.429821 3751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:28.433851 3751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 23:46:28.436107 3767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:28.437427 3751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:46:28.437611 3751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "71db0fe9dc9a47c8953a95ebd297e9b2"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "8727551b1ef0dcd250d4a13ebee2032d"
server_key_iv: "22fefc93355e18e59b3147c4d10bf1e3"
server_key_version: "encryptionkey@0"
I20260928 23:46:28.438249 3751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:28.454828 3751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:28.458106 3751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:28.458331 3751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:28.465654 3751 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:43455
I20260928 23:46:28.465971 3819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:43455 every 8 connection(s)
I20260928 23:46:28.466832 3751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:28.471318 3820 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:46:28.473037 3820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap starting.
I20260928 23:46:28.476946 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3751
I20260928 23:46:28.477432 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:45163
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:43455
--webserver_port=41891
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:28.490548 3820 log.cc:824] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:28.499977 3820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:46:28.500408 3820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap complete.
I20260928 23:46:28.504786 3820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:28.505213 3820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db0fe9dc9a47c8953a95ebd297e9b2, State: Initialized, Role: FOLLOWER
I20260928 23:46:28.505805 3820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:28.505929 3820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:28.506026 3820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:28.506155 3820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:46:28.508188 3820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:28.508546 3820 leader_election.cc:302] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71db0fe9dc9a47c8953a95ebd297e9b2; no voters:
I20260928 23:46:28.508921 3820 leader_election.cc:288] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:46:28.508987 3824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:46:28.509271 3824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 LEADER]: Becoming Leader. State: Replica: 71db0fe9dc9a47c8953a95ebd297e9b2, State: Running, Role: LEADER
I20260928 23:46:28.509673 3824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:28.510278 3820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:28.510661 3824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71db0fe9dc9a47c8953a95ebd297e9b2. Latest consensus state: current_term: 2 leader_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } } }
I20260928 23:46:28.510866 3824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:28.511229 3825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } } }
I20260928 23:46:28.511337 3825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:28.511569 3828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:28.516736 3828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:28.517562 3828 catalog_manager.cc:1259] Loaded cluster ID: 4e0900f3514f4f69959f319a8521ee3f
I20260928 23:46:28.517633 3828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:28.519106 3828 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 23:46:28.524451 3828 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 23:46:28.530239 3828 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:28.531085 3828 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Loaded TSK: 0
I20260928 23:46:28.532311 3828 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:46:28.595615 3822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:28.595919 3822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:28.595993 3822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:28.600114 3822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:28.600240 3822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:28.600337 3822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:46:28.605857 3822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:45163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=41891
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:28.606974 3822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:28.608003 3822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:28.618430 3848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:28.618445 3850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:28.618718 3847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:28.618579 3822 server_base.cc:1034] running on GCE node
I20260928 23:46:28.619112 3822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:28.619731 3822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:28.620931 3822 hybrid_clock.cc:648] HybridClock initialized: now 1790639188620904 us; error 43 us; skew 500 ppm
Sep 28 23:46:28 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:28.624457 3822 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:46:28.625651 3822 webserver.cc:460] Webserver started at http://127.21.112.65:41891/ using document root <none> and password file <none>
I20260928 23:46:28.626394 3822 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:28.626511 3822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:28.631861 3822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 23:46:28.635162 3857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:28.636873 3822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:46:28.637032 3822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "d0f121ee55bf4baa92f89b7306dc8c9c"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "3a1252dc8fb35028e11eb48e6f53faca"
server_key_iv: "766c3d075dde350fa49f62e6cdd1b5ff"
server_key_version: "encryptionkey@0"
I20260928 23:46:28.637863 3822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:28.679858 3822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:28.683445 3822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:28.683615 3822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:28.684209 3822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:28.685330 3864 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:46:28.689487 3822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:46:28.689611 3822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 23:46:28.689699 3822 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:46:28.691524 3822 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:46:28.691605 3822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:46:28.691864 3864 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap starting.
I20260928 23:46:28.703261 3822 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:45163
I20260928 23:46:28.703320 3971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:45163 every 8 connection(s)
I20260928 23:46:28.704391 3822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:46:28.705446 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3822
I20260928 23:46:28.705963 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:38549
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:43455
--webserver_port=37769
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:28.711800 3864 log.cc:824] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:46:28 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:28.722635 3864 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:46:28.723059 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.706712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:49797)
0928 23:46:28.707995 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:28.708064 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:28.709215 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:28.710666 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:28.710679 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:28.711254 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:28.712057 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.712086 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.713724 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.713727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.714586 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.714595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.715334 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.715339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.715523 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.716124 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:28.716141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:28.717684 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:28.719783 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:28.719788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:28.719801 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:28.720052 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:28.720398 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:28.720403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:28.720405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:28.720551 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:28.721259 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:28.721263 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:28.721509 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 23:46:28.721769 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 23:46:28.722110 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":148,"threads_started":1}
I20260928 23:46:28.723409 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.706665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:49797 (local address 127.21.112.126:43455)
0928 23:46:28.707835 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:28.707863 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:28.708217 (+ 354us) server_negotiation.cc:407] Connection header received
0928 23:46:28.709732 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:28.709748 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:28.710185 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:28.710557 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:28.712314 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.713479 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.714720 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.715189 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.715374 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.717879 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:28.717884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:28.717895 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:28.717922 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:28.719651 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:28.720177 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:28.720187 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:28.720191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:28.720302 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:28.720668 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:28.720671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:28.720673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:28.721022 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:28.721199 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:28.721884 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 23:46:28.722140 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":152,"threads_started":1}
I20260928 23:46:28.723229 3864 tablet_bootstrap.cc:492] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap complete.
I20260928 23:46:28.725374 3864 ts_tablet_manager.cc:1397] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20260928 23:46:28.726009 3972 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43455
I20260928 23:46:28.726250 3972 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:28.726742 3972 heartbeater.cc:511] Master 127.21.112.126:43455 requested a full tablet report, sending...
I20260928 23:46:28.729110 3779 ts_manager.cc:194] Registered new tserver with Master: d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163)
I20260928 23:46:28.728718 3864 raft_consensus.cc:348] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.729442 3864 raft_consensus.cc:729] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f121ee55bf4baa92f89b7306dc8c9c, State: Initialized, Role: FOLLOWER
I20260928 23:46:28.730072 3864 consensus_queue.cc:260] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:28.731227 3864 ts_tablet_manager.cc:1428] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 23:46:28.732947 3779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:49797
I20260928 23:46:28.735999 3972 heartbeater.cc:503] Master 127.21.112.126:43455 was elected leader, sending a full tablet report...
I20260928 23:46:28.738976 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.732470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:45163 (local address 127.0.0.1:56148)
0928 23:46:28.734067 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:28.734124 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:28.734294 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:28.735155 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:28.735161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:28.735184 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:28.735318 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.735324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.736303 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.736306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.737602 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.737614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.738410 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.738414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.738520 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.738556 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:46:28.738707 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 23:46:28.738837 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":119,"threads_started":1}
I20260928 23:46:28.738976 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.732533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.21.112.65:45163)
0928 23:46:28.733785 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:28.733798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:28.734228 (+ 430us) server_negotiation.cc:407] Connection header received
0928 23:46:28.734469 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:28.734473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:28.734989 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:28.735145 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:28.735428 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.736175 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.737731 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.738278 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.738404 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.738544 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:28.738737 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 23:46:28.738830 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1144,"thread_start_us":107,"threads_started":1}
I20260928 23:46:28.740253 3906 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a973dbb479b444baf65d4f5cff0831f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:28 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:56148
I20260928 23:46:28.741340 3981 tablet_replica.cc:330] stopping tablet replica
I20260928 23:46:28.741729 3981 raft_consensus.cc:2229] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:28.742043 3981 raft_consensus.cc:2258] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:28.743197 3981 ts_tablet_manager.cc:1905] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:28.747189 3981 ts_tablet_manager.cc:1918] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:28.747308 3981 log.cc:1196] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/4a973dbb479b444baf65d4f5cff0831f
I20260928 23:46:28.747628 3981 ts_tablet_manager.cc:1939] T 4a973dbb479b444baf65d4f5cff0831f P d0f121ee55bf4baa92f89b7306dc8c9c: Deleting consensus metadata
I20260928 23:46:28.749666 3771 catalog_manager.cc:4841] TS d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163): tablet 4a973dbb479b444baf65d4f5cff0831f (table test-table [id=efae263e9a6c47469a0bf9af07623dcf]) successfully deleted
W20260928 23:46:28.836862 3976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:28.837129 3976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:28.837173 3976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:28.841244 3976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:28.841307 3976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:28.841388 3976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:46:28.847092 3976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=37769
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:28.848218 3976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:28.849287 3976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:46:28.859225 3976 server_base.cc:1034] running on GCE node
W20260928 23:46:28.859138 3987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:28.859158 3986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:28.859133 3989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:28.859838 3976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:28.860538 3976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:28.861714 3976 hybrid_clock.cc:648] HybridClock initialized: now 1790639188861697 us; error 29 us; skew 500 ppm
Sep 28 23:46:28 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:28.864568 3976 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:46:28.865410 3976 webserver.cc:460] Webserver started at http://127.21.112.66:37769/ using document root <none> and password file <none>
I20260928 23:46:28.866019 3976 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:28.866091 3976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:28.869897 3976 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 23:46:28.871830 3996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:28.873116 3976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:46:28.873251 3976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "4e0439409b474559acdcdc933b4a8a88"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "5fcd74a4f7c53ce7decc941cda9d737a"
server_key_iv: "63e2bf223e01c84069efb4751ef9a6e1"
server_key_version: "encryptionkey@0"
I20260928 23:46:28.873868 3976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:28.891036 3976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:28.894855 3976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:28.895072 3976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:28.895712 3976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:28.896605 3976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:28.896662 3976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:28.896703 3976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:28.896744 3976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:28.906038 3976 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:38549
I20260928 23:46:28.906054 4109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:38549 every 8 connection(s)
I20260928 23:46:28.907094 3976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:46:28.915088 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 3976
I20260928 23:46:28.915529 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:42177
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:43455
--webserver_port=35183
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:46:28 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639188, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:28.922035 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.908945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:56941 (local address 127.21.112.126:43455)
0928 23:46:28.909091 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:28.909097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:28.910041 (+ 944us) server_negotiation.cc:407] Connection header received
0928 23:46:28.911521 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:28.911523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:28.911576 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:28.911705 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:28.912960 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.913497 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.914714 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.915045 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.915163 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.917907 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:28.917910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:28.917912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:28.917936 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:28.919754 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:28.920243 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:28.920249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:28.920254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:28.920352 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:28.920694 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:28.920698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:28.920700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:28.920976 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:28.921093 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:28.921617 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 23:46:28.921784 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:46:28.922827 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:28.909373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:56941)
0928 23:46:28.909887 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:28.909931 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:28.911321 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:28.911904 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:28.911911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:28.912276 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:28.912812 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.912835 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.913691 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.913694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.914609 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:28.914618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:28.915231 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:28.915237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:28.915407 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:28.915993 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:28.916010 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:28.917746 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:28.919862 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:28.919867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:28.919881 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:28.920138 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:28.920447 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:28.920449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:28.920450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:28.920584 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:28.921093 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:28.921101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:28.921347 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 23:46:28.921509 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 23:46:28.921922 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":172,"threads_started":1}
I20260928 23:46:28.924120 4110 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43455
I20260928 23:46:28.924366 4110 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:28.924885 4110 heartbeater.cc:511] Master 127.21.112.126:43455 requested a full tablet report, sending...
I20260928 23:46:28.926048 3779 ts_manager.cc:194] Registered new tserver with Master: 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
I20260928 23:46:28.926687 3779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:56941
W20260928 23:46:29.040491 4113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:29.040791 4113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:29.040838 4113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:29.045228 4113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:29.045300 4113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:29.045364 4113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:46:29.051218 4113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:42177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=35183
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:29.052328 4113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:29.053345 4113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:29.063566 4118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.063763 4121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.063566 4119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:29.063895 4113 server_base.cc:1034] running on GCE node
I20260928 23:46:29.064442 4113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:29.065104 4113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:29.066295 4113 hybrid_clock.cc:648] HybridClock initialized: now 1790639189066282 us; error 36 us; skew 500 ppm
Sep 28 23:46:29 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:29.069100 4113 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:46:29.070174 4113 webserver.cc:460] Webserver started at http://127.21.112.67:35183/ using document root <none> and password file <none>
I20260928 23:46:29.070760 4113 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:29.070863 4113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:29.074651 4113 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 23:46:29.076709 4128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.078019 4113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:46:29.078156 4113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "4e75d7000058444ca07fc3bd1540c506"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "b322c0ec90c09137bd921d4f1792b633"
server_key_iv: "c165daf87bb0275d3d9dc11e309c74bf"
server_key_version: "encryptionkey@0"
I20260928 23:46:29.078693 4113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:29.094743 4113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:29.097961 4113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:29.098232 4113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:29.098850 4113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:29.099716 4113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:29.099785 4113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.099857 4113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:29.099912 4113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.109441 4113 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:42177
I20260928 23:46:29.109483 4241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:42177 every 8 connection(s)
I20260928 23:46:29.110581 4113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:46:29.114173 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4113
Sep 28 23:46:29 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:29.124115 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.114981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.21.112.126:43455)
0928 23:46:29.115270 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.115274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.115331 (+ 57us) server_negotiation.cc:407] Connection header received
0928 23:46:29.115529 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.115532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.115581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.115697 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:29.116687 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.117265 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.118319 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.118674 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.118778 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.120337 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:29.120340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:29.120342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:29.120366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:29.122346 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:29.122787 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:29.122792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:29.122795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:29.122899 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:29.123216 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:29.123220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:29.123222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:29.123398 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:29.123631 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.123802 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.123904 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":70,"threads_started":1}
I20260928 23:46:29.127700 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.112556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:54879 (local address 127.21.112.126:43455)
0928 23:46:29.112718 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.112721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.113805 (+ 1084us) server_negotiation.cc:407] Connection header received
0928 23:46:29.116112 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.116115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.116161 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.116219 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:29.117934 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.118665 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.120431 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.120788 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.120867 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.124258 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:29.124263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:29.124265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:29.124288 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:29.125823 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:29.126332 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:29.126334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:29.126335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:29.126403 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:29.126761 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:29.126763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:29.126765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:29.126910 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:29.126979 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.127515 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.127597 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:46:29.128690 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.112961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:54879)
0928 23:46:29.113506 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.113656 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:29.115802 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:29.116489 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:29.116500 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:29.117004 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:29.117776 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.117796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.118804 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.118807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.120317 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.120326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.120944 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.120949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.121128 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.121779 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:29.121804 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:29.124088 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:29.125934 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:29.125940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:29.125955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:29.126233 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:29.126517 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:29.126520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:29.126522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:29.126668 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:29.127034 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:29.127039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:29.127304 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 23:46:29.127458 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 23:46:29.127706 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":121,"threads_started":1}
I20260928 23:46:29.129803 4242 heartbeater.cc:347] Connected to a master server at 127.21.112.126:43455
I20260928 23:46:29.130052 4242 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:29.130551 4242 heartbeater.cc:511] Master 127.21.112.126:43455 requested a full tablet report, sending...
I20260928 23:46:29.131563 3779 ts_manager.cc:194] Registered new tserver with Master: 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177)
I20260928 23:46:29.132141 3779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:54879
I20260928 23:46:29.137807 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:46:29.141919 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.138421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.21.112.126:43455)
0928 23:46:29.138536 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.138540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.138751 (+ 211us) server_negotiation.cc:407] Connection header received
0928 23:46:29.138883 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.138885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.138976 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.139057 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:46:29.139298 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.139786 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.140600 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.140808 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.140885 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.141158 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:46:29.141524 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:46:29.141736 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.141773 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.141821 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:46:29.142547 3779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:46:29.146221 3779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:46:29.153815 3906 tablet_service.cc:1467] Processing CreateTablet for tablet 2c262ab4b58e4f8ca61d7f131438c711 (DEFAULT_TABLE table=test-table [id=08d57704b369470fa6ede64d0cfd3de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:29.154547 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c262ab4b58e4f8ca61d7f131438c711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:29.157159 3864 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap starting.
I20260928 23:46:29.158118 3864 tablet_bootstrap.cc:654] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:29.159217 3864 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: No bootstrap required, opened a new log
I20260928 23:46:29.159222 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.152754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:42177 (local address 127.0.0.1:32882)
0928 23:46:29.153024 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.153039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:29.153140 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:29.154259 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:29.154262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:29.154279 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:29.154606 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.154616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.156053 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.156056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.157022 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.157032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.158817 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.158821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.158909 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.158935 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:46:29.158987 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:46:29.159064 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":260,"spinlock_wait_cycles":1792}
I20260928 23:46:29.159304 3864 ts_tablet_manager.cc:1397] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:46:29.159684 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.152930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38549 (local address 127.0.0.1:41992)
0928 23:46:29.153388 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.153399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:29.153466 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:29.154295 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:29.154298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:29.154317 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:29.154440 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.154447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.156053 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.156056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.158163 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.158173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.159266 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.159270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.159367 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.159394 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 23:46:29.159479 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:46:29.159545 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":902,"thread_start_us":100,"threads_started":1}
I20260928 23:46:29.159729 3864 raft_consensus.cc:348] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.159844 3864 raft_consensus.cc:374] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:29.159884 3864 raft_consensus.cc:729] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f121ee55bf4baa92f89b7306dc8c9c, State: Initialized, Role: FOLLOWER
I20260928 23:46:29.159986 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.152939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.21.112.67:42177)
0928 23:46:29.153748 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.153760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.153779 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:46:29.153838 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.153842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.154081 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.154386 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:29.154756 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.155894 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.157646 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.158682 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.159512 (+ 830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.159648 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.159691 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.159797 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":175,"threads_started":1}
I20260928 23:46:29.160009 3864 consensus_queue.cc:260] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.160131 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.153343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.21.112.66:38549)
0928 23:46:29.153706 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.153719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.153744 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:46:29.153830 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.153835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.154150 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.154308 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:29.154563 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.155895 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.158305 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.159134 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.159250 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.159407 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.159861 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.159975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260928 23:46:29.160290 3864 ts_tablet_manager.cc:1428] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:46:29.163233 4044 tablet_service.cc:1467] Processing CreateTablet for tablet 2c262ab4b58e4f8ca61d7f131438c711 (DEFAULT_TABLE table=test-table [id=08d57704b369470fa6ede64d0cfd3de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:29.163226 4176 tablet_service.cc:1467] Processing CreateTablet for tablet 2c262ab4b58e4f8ca61d7f131438c711 (DEFAULT_TABLE table=test-table [id=08d57704b369470fa6ede64d0cfd3de3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:29.164368 4176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c262ab4b58e4f8ca61d7f131438c711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:29.164366 4044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c262ab4b58e4f8ca61d7f131438c711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:29.173751 4255 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Bootstrap starting.
I20260928 23:46:29.176054 4255 tablet_bootstrap.cc:654] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:29.177174 4255 log.cc:824] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:29.177639 4256 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Bootstrap starting.
I20260928 23:46:29.179266 4255 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: No bootstrap required, opened a new log
I20260928 23:46:29.179471 4255 ts_tablet_manager.cc:1397] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:46:29.179912 4256 tablet_bootstrap.cc:654] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:29.180575 4256 log.cc:824] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:29.182941 4256 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: No bootstrap required, opened a new log
I20260928 23:46:29.182878 4255 raft_consensus.cc:348] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.183143 4255 raft_consensus.cc:374] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:29.183179 4256 ts_tablet_manager.cc:1397] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 23:46:29.183245 4255 raft_consensus.cc:729] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e75d7000058444ca07fc3bd1540c506, State: Initialized, Role: FOLLOWER
I20260928 23:46:29.183712 4255 consensus_queue.cc:260] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.184422 4242 heartbeater.cc:503] Master 127.21.112.126:43455 was elected leader, sending a full tablet report...
I20260928 23:46:29.184695 4255 ts_tablet_manager.cc:1428] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:46:29.186941 4256 raft_consensus.cc:348] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.187242 4256 raft_consensus.cc:374] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:29.187304 4256 raft_consensus.cc:729] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0439409b474559acdcdc933b4a8a88, State: Initialized, Role: FOLLOWER
I20260928 23:46:29.187793 4256 consensus_queue.cc:260] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.188824 4256 ts_tablet_manager.cc:1428] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260928 23:46:29.189186 4110 heartbeater.cc:503] Master 127.21.112.126:43455 was elected leader, sending a full tablet report...
W20260928 23:46:29.206863 3973 tablet.cc:2369] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:46:29.250135 4259 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:29.250367 4259 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.251588 4259 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549), d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163)
I20260928 23:46:29.255868 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.251898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38549 (local address 127.21.112.67:42859)
0928 23:46:29.252074 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.252094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:29.252238 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:29.252599 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:29.252602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:29.252640 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:29.252769 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.252775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.253683 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.253687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.254577 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.254587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.255370 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.255376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.255512 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.255540 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:46:29.255627 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:46:29.255711 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"spinlock_wait_cycles":12544}
I20260928 23:46:29.255867 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.252057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:42859 (local address 127.21.112.66:38549)
0928 23:46:29.252195 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.252203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.252226 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:46:29.252422 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.252426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.252477 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.252580 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:29.252908 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.253488 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.254734 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.255235 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.255374 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.255457 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.255656 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.255741 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:46:29.256636 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.251898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:45163 (local address 127.21.112.67:59433)
0928 23:46:29.252480 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.252492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:29.252643 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:29.252918 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:29.252921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:29.252936 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:29.253063 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.253069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.253865 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.253869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.254651 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:29.254659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.256224 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.256231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:29.256346 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.256376 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:46:29.256455 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:46:29.256512 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":149,"spinlock_wait_cycles":2944,"thread_start_us":109,"threads_started":1}
I20260928 23:46:29.256734 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.252052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:59433 (local address 127.21.112.65:45163)
0928 23:46:29.252355 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.252371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.252577 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:46:29.252755 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.252758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.252811 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.252908 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:29.253175 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.253758 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.254779 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.255345 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.255705 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.255752 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.256499 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.256584 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260928 23:46:29.256639 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262ab4b58e4f8ca61d7f131438c711" candidate_uuid: "4e75d7000058444ca07fc3bd1540c506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0439409b474559acdcdc933b4a8a88" is_pre_election: true
I20260928 23:46:29.257005 4064 raft_consensus.cc:2454] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e75d7000058444ca07fc3bd1540c506 in term 0.
I20260928 23:46:29.257371 3926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262ab4b58e4f8ca61d7f131438c711" candidate_uuid: "4e75d7000058444ca07fc3bd1540c506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" is_pre_election: true
I20260928 23:46:29.257697 3926 raft_consensus.cc:2454] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e75d7000058444ca07fc3bd1540c506 in term 0.
I20260928 23:46:29.257732 4129 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0439409b474559acdcdc933b4a8a88, 4e75d7000058444ca07fc3bd1540c506; no voters:
I20260928 23:46:29.258104 4259 raft_consensus.cc:2781] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:46:29.258183 4259 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:46:29.258216 4259 raft_consensus.cc:3037] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:29.259382 4259 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.259884 4259 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [CANDIDATE]: Term 1 election: Requested vote from peers 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549), d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163)
I20260928 23:46:29.260471 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262ab4b58e4f8ca61d7f131438c711" candidate_uuid: "4e75d7000058444ca07fc3bd1540c506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0439409b474559acdcdc933b4a8a88"
I20260928 23:46:29.260471 3926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c262ab4b58e4f8ca61d7f131438c711" candidate_uuid: "4e75d7000058444ca07fc3bd1540c506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c"
I20260928 23:46:29.260625 4064 raft_consensus.cc:3037] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:29.260625 3926 raft_consensus.cc:3037] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:29.262091 3926 raft_consensus.cc:2454] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e75d7000058444ca07fc3bd1540c506 in term 1.
I20260928 23:46:29.262091 4064 raft_consensus.cc:2454] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e75d7000058444ca07fc3bd1540c506 in term 1.
I20260928 23:46:29.262463 4132 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e75d7000058444ca07fc3bd1540c506, d0f121ee55bf4baa92f89b7306dc8c9c; no voters:
I20260928 23:46:29.262697 4259 raft_consensus.cc:2781] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:29.262979 4259 raft_consensus.cc:686] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 1 LEADER]: Becoming Leader. State: Replica: 4e75d7000058444ca07fc3bd1540c506, State: Running, Role: LEADER
I20260928 23:46:29.263365 4259 consensus_queue.cc:237] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.267053 3783 catalog_manager.cc:5495] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67). New cstate: current_term: 1 leader_uuid: "4e75d7000058444ca07fc3bd1540c506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } health_report { overall_health: UNKNOWN } } }
I20260928 23:46:29.290072 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.286079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.21.112.67:42177)
0928 23:46:29.286243 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:29.286248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:29.286264 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:46:29.286445 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:29.286448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:29.286506 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:29.286632 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:46:29.286899 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.287462 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.288436 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:29.288674 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:29.288796 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:29.289141 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:46:29.289621 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:46:29.289767 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:29.289836 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:46:29.289897 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:46:29.296607 4064 raft_consensus.cc:1261] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Refusing update from remote peer 4e75d7000058444ca07fc3bd1540c506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:29.296604 3926 raft_consensus.cc:1261] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Refusing update from remote peer 4e75d7000058444ca07fc3bd1540c506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:29.297494 4263 consensus_queue.cc:1035] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:29.297757 4259 consensus_queue.cc:1035] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:29.305918 4267 mvcc.cc:205] Tried to move back new op lower bound from 7334458119350341632 to 7334458119226294272. Current Snapshot: MvccSnapshot[applied={T|T < 7334458119350341632}]
I20260928 23:46:29.313711 3783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:29.313910 3783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:29.316589 3783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Sending DeleteTablet for 3 replicas of tablet 2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:29.317267 4176 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c262ab4b58e4f8ca61d7f131438c711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:29 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:32882
I20260928 23:46:29.317414 4044 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c262ab4b58e4f8ca61d7f131438c711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:29 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:41992
I20260928 23:46:29.317468 3906 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c262ab4b58e4f8ca61d7f131438c711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:29 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:56148
I20260928 23:46:29.317926 4277 tablet_replica.cc:330] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: stopping tablet replica
I20260928 23:46:29.317945 4278 tablet_replica.cc:330] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: stopping tablet replica
I20260928 23:46:29.318027 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3822
I20260928 23:46:29.318311 4278 raft_consensus.cc:2229] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:29.318744 4278 raft_consensus.cc:2258] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:29.318953 4277 raft_consensus.cc:2229] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:46:29.319799 4277 raft_consensus.cc:2258] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:29.320547 4278 ts_tablet_manager.cc:1905] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:29.321302 4277 ts_tablet_manager.cc:1905] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:29.323781 4278 ts_tablet_manager.cc:1918] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:29.323915 4278 log.cc:1196] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:29.324334 4278 ts_tablet_manager.cc:1939] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e0439409b474559acdcdc933b4a8a88: Deleting consensus metadata
I20260928 23:46:29.325038 4277 ts_tablet_manager.cc:1918] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:29.325217 4277 log.cc:1196] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:29.325655 4277 ts_tablet_manager.cc:1939] T 2c262ab4b58e4f8ca61d7f131438c711 P 4e75d7000058444ca07fc3bd1540c506: Deleting consensus metadata
I20260928 23:46:29.325822 3768 catalog_manager.cc:4841] TS 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): tablet 2c262ab4b58e4f8ca61d7f131438c711 (table test-table [id=08d57704b369470fa6ede64d0cfd3de3]) successfully deleted
W20260928 23:46:29.326344 3771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:45163 (error 108)
I20260928 23:46:29.326870 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3976
I20260928 23:46:29.329294 3770 catalog_manager.cc:4841] TS 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): tablet 2c262ab4b58e4f8ca61d7f131438c711 (table test-table [id=08d57704b369470fa6ede64d0cfd3de3]) successfully deleted
I20260928 23:46:29.334285 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 4113
I20260928 23:46:29.335099 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.334512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:45163 (local address 127.0.0.1:56164)
0928 23:46:29.334911 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.334992 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:45163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
W20260928 23:46:29.335359 3771 catalog_manager.cc:4584] TS d0f121ee55bf4baa92f89b7306dc8c9c (127.21.112.65:45163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c262ab4b58e4f8ca61d7f131438c711: Network error: Client connection negotiation failed: client connection to 127.21.112.65:45163: connect: Connection refused (error 111)
I20260928 23:46:29.340832 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 3751
I20260928 23:46:29.348155 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43455
--webserver_interface=127.21.112.126
--webserver_port=45485
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:29.462466 4280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:29.462798 4280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:29.462889 4280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:29.466928 4280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:29.467017 4280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:29.467056 4280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:29.467120 4280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:29.467157 4280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:29.472399 4280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=45485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:29.473516 4280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:29.474599 4280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:46:29.484141 4280 server_base.cc:1034] running on GCE node
W20260928 23:46:29.484089 4285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.484127 4288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.484089 4286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:29.484951 4280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:29.486217 4280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:29.487383 4280 hybrid_clock.cc:648] HybridClock initialized: now 1790639189487366 us; error 32 us; skew 500 ppm
Sep 28 23:46:29 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:29.490370 4280 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:29.491225 4280 webserver.cc:460] Webserver started at http://127.21.112.126:45485/ using document root <none> and password file <none>
I20260928 23:46:29.491762 4280 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:29.491820 4280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:29.495797 4280 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:46:29.497961 4295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.499266 4280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:46:29.499398 4280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "71db0fe9dc9a47c8953a95ebd297e9b2"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "8727551b1ef0dcd250d4a13ebee2032d"
server_key_iv: "22fefc93355e18e59b3147c4d10bf1e3"
server_key_version: "encryptionkey@0"
I20260928 23:46:29.499946 4280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:29.515334 4280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:29.518808 4280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:29.519032 4280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:29.526347 4280 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:43455
I20260928 23:46:29.526649 4347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:43455 every 8 connection(s)
I20260928 23:46:29.527495 4280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:29.532039 4348 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:46:29.533783 4348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap starting.
I20260928 23:46:29.535063 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4280
I20260928 23:46:29.535547 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:45163
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:43455
--webserver_port=41891
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:29.551664 4348 log.cc:824] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:29.563644 4348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:46:29.564069 4348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2: Bootstrap complete.
I20260928 23:46:29.568203 4348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:29.568643 4348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71db0fe9dc9a47c8953a95ebd297e9b2, State: Initialized, Role: FOLLOWER
I20260928 23:46:29.569200 4348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:29.569350 4348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:29.569440 4348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:29.569573 4348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:46:29.571488 4348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:29.571830 4348 leader_election.cc:302] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71db0fe9dc9a47c8953a95ebd297e9b2; no voters:
I20260928 23:46:29.572155 4348 leader_election.cc:288] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:46:29.572301 4352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:46:29.572556 4352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [term 3 LEADER]: Becoming Leader. State: Replica: 71db0fe9dc9a47c8953a95ebd297e9b2, State: Running, Role: LEADER
I20260928 23:46:29.572875 4352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } }
I20260928 23:46:29.573395 4348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:29.574333 4353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } } }
I20260928 23:46:29.574461 4353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:29.574781 4354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71db0fe9dc9a47c8953a95ebd297e9b2. Latest consensus state: current_term: 3 leader_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71db0fe9dc9a47c8953a95ebd297e9b2" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43455 } } }
I20260928 23:46:29.574867 4354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71db0fe9dc9a47c8953a95ebd297e9b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:29.575171 4361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:29.580371 4361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:29.581512 4361 catalog_manager.cc:1259] Loaded cluster ID: 4e0900f3514f4f69959f319a8521ee3f
I20260928 23:46:29.581616 4361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 23:46:29.582630 4361 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f88868de9ad google::LogMessage::Fail() at ??:0
@ 0x7f88868e188a google::LogMessage::SendToLog() at ??:0
@ 0x7f88868de4de google::LogMessage::Flush() at ??:0
@ 0x7f88868dfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f889121dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f889121ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f889121b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f889125f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8888b303d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8888b30c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8888b323f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8888b21512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f888a1656db start_thread at ??:0
@ 0x7f88853dc61f clone at ??:0
W20260928 23:46:29.668866 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:29.669183 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:29.669265 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:29.673384 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:29.673473 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:29.673625 4350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:46:29.679075 4350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:45163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=41891
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:29.680241 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:29.681284 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:29.691421 4392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.691496 4390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.691460 4389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:29.691882 4350 server_base.cc:1034] running on GCE node
I20260928 23:46:29.692324 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:29.692989 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:29.694185 4350 hybrid_clock.cc:648] HybridClock initialized: now 1790639189694156 us; error 46 us; skew 500 ppm
Sep 28 23:46:29 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:29.697014 4350 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:46:29.698035 4350 webserver.cc:460] Webserver started at http://127.21.112.65:41891/ using document root <none> and password file <none>
I20260928 23:46:29.698611 4350 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:29.698674 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:29.702765 4350 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:46:29.705096 4399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.706663 4350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:46:29.706820 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "d0f121ee55bf4baa92f89b7306dc8c9c"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "3a1252dc8fb35028e11eb48e6f53faca"
server_key_iv: "766c3d075dde350fa49f62e6cdd1b5ff"
server_key_version: "encryptionkey@0"
I20260928 23:46:29.707454 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:29.724360 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:29.728044 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:29.728260 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:29.728919 4350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:29.730146 4406 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:46:29.734581 4350 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:46:29.734652 4350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 23:46:29.734724 4350 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:46:29.736418 4350 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:46:29.736475 4350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:46:29.736742 4406 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap starting.
I20260928 23:46:29.747537 4350 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:45163
I20260928 23:46:29.747562 4513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:45163 every 8 connection(s)
I20260928 23:46:29.748948 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:46:29.752894 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4350
I20260928 23:46:29.753404 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:38549
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:43455
--webserver_port=37769
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:46:29.758495 4406 log.cc:824] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:29.769269 4406 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:46:29.769752 4406 tablet_bootstrap.cc:492] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Bootstrap complete.
I20260928 23:46:29.771209 4406 ts_tablet_manager.cc:1397] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.012s
I20260928 23:46:29.774547 4406 raft_consensus.cc:348] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.775107 4406 raft_consensus.cc:729] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f121ee55bf4baa92f89b7306dc8c9c, State: Initialized, Role: FOLLOWER
I20260928 23:46:29.775667 4406 consensus_queue.cc:260] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:29.776788 4406 ts_tablet_manager.cc:1428] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260928 23:46:29.842530 4516 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:46:29.751532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:55913)
0928 23:46:29.752182 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.752252 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:29.755859 (+ 3607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:29.841725 (+ 85866us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":182,"threads_started":1}
W20260928 23:46:29.843226 4514 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:43455 (0 consecutive failures): Network error: Failed to ping master at 127.21.112.126:43455: Client connection negotiation failed: client connection to 127.21.112.126:43455: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:46:29.844110 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.843803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:43697)
0928 23:46:29.843950 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.844035 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 23:46:29.844836 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.844625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:54267)
0928 23:46:29.844731 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.844779 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:46:29.845090 4514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:46:29.879041 4517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:29.879352 4517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:29.879400 4517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:29.883846 4517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:29.883925 4517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:29.884037 4517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:46:29.890141 4517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=37769
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:29.891352 4517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:29.892463 4517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:29.902980 4525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:29.903021 4527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:29.903003 4517 server_base.cc:1034] running on GCE node
W20260928 23:46:29.903005 4524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:29.903676 4517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:29.904420 4517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:29.905632 4517 hybrid_clock.cc:648] HybridClock initialized: now 1790639189905600 us; error 50 us; skew 500 ppm
Sep 28 23:46:29 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:29.908758 4517 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:46:29.910029 4517 webserver.cc:460] Webserver started at http://127.21.112.66:37769/ using document root <none> and password file <none>
I20260928 23:46:29.910725 4517 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:29.910830 4517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:29.915282 4517 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:46:29.917529 4534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.918895 4517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:46:29.919037 4517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "4e0439409b474559acdcdc933b4a8a88"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "5fcd74a4f7c53ce7decc941cda9d737a"
server_key_iv: "63e2bf223e01c84069efb4751ef9a6e1"
server_key_version: "encryptionkey@0"
I20260928 23:46:29.919607 4517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:29.947865 4517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:29.951066 4517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:29.951320 4517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:29.951936 4517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:29.952762 4517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:29.952833 4517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.952905 4517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:29.952950 4517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:29.962625 4517 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:38549
I20260928 23:46:29.962667 4647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:38549 every 8 connection(s)
I20260928 23:46:29.963653 4517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:46:29.967780 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.965863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:58839)
0928 23:46:29.966362 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.966480 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
W20260928 23:46:29.968321 4648 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:43455 (0 consecutive failures): Network error: Failed to ping master at 127.21.112.126:43455: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
I20260928 23:46:29.968961 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.968750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:52365)
0928 23:46:29.968859 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.968908 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 23:46:29.969622 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:29.969390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:52685)
0928 23:46:29.969485 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:29.969524 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 23:46:29.969781 4648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:46:29.973951 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4517
I20260928 23:46:29.974347 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:42177
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:43455
--webserver_port=35183
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:30.089033 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:30.089298 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:30.089340 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:30.093358 4651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:30.093432 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:30.093490 4651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:46:30.098901 4651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:42177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=35183
--tserver_master_addrs=127.21.112.126:43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:30.100044 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:30.101097 4651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:30.111207 4656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:30.111191 4657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:30.111191 4659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:30.111730 4651 server_base.cc:1034] running on GCE node
I20260928 23:46:30.112195 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:30.112836 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:30.114024 4651 hybrid_clock.cc:648] HybridClock initialized: now 1790639190113998 us; error 38 us; skew 500 ppm
Sep 28 23:46:30 dist-test-slave-c3s8 krb5kdc[3202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639190, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:30.116794 4651 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:46:30.117848 4651 webserver.cc:460] Webserver started at http://127.21.112.67:35183/ using document root <none> and password file <none>
I20260928 23:46:30.118391 4651 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:30.118449 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:30.122354 4651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:46:30.124598 4666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:30.125849 4651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:46:30.125975 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "4e75d7000058444ca07fc3bd1540c506"
format_stamp: "Formatted at 2026-09-28 23:46:27 on dist-test-slave-c3s8"
server_key: "b322c0ec90c09137bd921d4f1792b633"
server_key_iv: "c165daf87bb0275d3d9dc11e309c74bf"
server_key_version: "encryptionkey@0"
I20260928 23:46:30.126619 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:30.145857 4651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:30.149647 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:30.149879 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:30.150543 4651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:30.151430 4651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:30.151479 4651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:30.151553 4651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:30.151597 4651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:30.160933 4651 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:42177
I20260928 23:46:30.160940 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:42177 every 8 connection(s)
I20260928 23:46:30.162081 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:46:30.166288 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:30.164401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:36403)
0928 23:46:30.164876 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:30.165001 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
W20260928 23:46:30.166767 4780 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:43455 (0 consecutive failures): Network error: Failed to ping master at 127.21.112.126:43455: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
I20260928 23:46:30.167359 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:30.167151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:46777)
0928 23:46:30.167255 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:30.167304 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:46:30.168015 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:30.167845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:50289)
0928 23:46:30.167934 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:30.167973 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 23:46:30.168241 4780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:46:30.171423 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4651
I20260928 23:46:30.846161 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:30.845727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:38021)
0928 23:46:30.845999 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:30.846053 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260928 23:46:30.970705 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:30.970313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:34333)
0928 23:46:30.970542 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:30.970599 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260928 23:46:31.169325 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.168828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:55155)
0928 23:46:31.169159 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:31.169221 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260928 23:46:31.437155 4787 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:31.437434 4787 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:31.438704 4787 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177), 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
Sep 28 23:46:31 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.67@KRBTEST.COM
Sep 28 23:46:31 dist-test-slave-c3s8 krb5kdc[3202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639189, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.66@KRBTEST.COM
I20260928 23:46:31.453099 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.438981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:42177 (local address 127.21.112.65:45575)
0928 23:46:31.439307 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:31.439324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:31.439438 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:31.441304 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:31.441310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:31.441933 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:31.442414 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:31.442429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.443727 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.443732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:31.444626 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:31.444635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.445356 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.445359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:31.445661 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:31.446339 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:31.446355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:31.448000 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:31.450463 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:31.450472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:31.450487 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:31.450770 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:31.451122 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:31.451126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:31.451127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:31.451325 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:31.452356 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:31.452361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:31.452584 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 23:46:31.452740 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 23:46:31.452962 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":121,"spinlock_wait_cycles":3456,"thread_start_us":81,"threads_started":1}
I20260928 23:46:31.453424 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.439220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45575 (local address 127.21.112.67:42177)
0928 23:46:31.439707 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:31.439725 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:31.439758 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:46:31.440405 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:31.440411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:31.440884 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:31.441216 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:31.442586 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.443576 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.444750 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.445230 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.445403 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:31.448238 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:31.448243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:31.448256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:31.448284 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:31.450338 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:31.450892 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:31.450903 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:31.450907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:31.451019 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:31.451436 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:31.451442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:31.451447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:31.452225 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:31.452375 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:31.452881 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 23:46:31.453228 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":89,"threads_started":1}
I20260928 23:46:31.453766 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.438981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38549 (local address 127.21.112.65:49153)
0928 23:46:31.439389 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:31.439403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:31.439475 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:31.441305 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:31.441312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:31.441935 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:31.442414 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:31.442430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.443783 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.443790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:31.444628 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:31.444638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.445393 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.445397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:31.445663 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:31.446407 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:31.446421 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:31.448393 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:31.450849 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:31.450856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:31.450858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:31.451078 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:31.451469 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:31.451473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:31.451478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:31.451566 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:31.453268 (+ 1702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:31.453271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:31.453356 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 23:46:31.453459 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 23:46:31.453609 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":187,"spinlock_wait_cycles":10496,"thread_start_us":85,"threads_started":1}
I20260928 23:46:31.454209 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.439375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:49153 (local address 127.21.112.66:38549)
0928 23:46:31.439671 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:31.439695 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:31.439732 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:46:31.440585 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:31.440593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:31.441014 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:31.441370 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:31.442612 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.443648 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.444757 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:31.445259 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:31.445444 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:31.448611 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:31.448616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:31.448653 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:31.448694 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:31.450721 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:31.451219 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:31.451225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:31.451231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:31.451360 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:31.451670 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:31.451674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:31.451676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:31.452225 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:31.452375 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:31.453640 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0928 23:46:31.453947 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":59,"threads_started":1}
W20260928 23:46:31.455946 4402 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
W20260928 23:46:31.456264 4400 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:31.456347 4400 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0f121ee55bf4baa92f89b7306dc8c9c; no voters: 4e0439409b474559acdcdc933b4a8a88, 4e75d7000058444ca07fc3bd1540c506
I20260928 23:46:31.456619 4787 raft_consensus.cc:2726] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:46:31.847527 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.847093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:51473)
0928 23:46:31.847298 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:31.847415 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 23:46:31.971858 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:31.971440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:48263)
0928 23:46:31.971710 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:31.971767 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
I20260928 23:46:32.170583 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:32.170184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:44531)
0928 23:46:32.170431 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:32.170490 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260928 23:46:32.848855 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:32.848367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:48381)
0928 23:46:32.848702 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:32.848757 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":74,"threads_started":1}
I20260928 23:46:32.973002 4795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:32.972599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:43171)
0928 23:46:32.972855 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:32.972911 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260928 23:46:33.171860 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:33.171392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:42645)
0928 23:46:33.171693 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:33.171759 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260928 23:46:33.516599 4797 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:33.516726 4797 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:33.517120 4797 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177), 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
W20260928 23:46:33.517973 4402 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
W20260928 23:46:33.518061 4400 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:33.518168 4400 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0f121ee55bf4baa92f89b7306dc8c9c; no voters: 4e0439409b474559acdcdc933b4a8a88, 4e75d7000058444ca07fc3bd1540c506
I20260928 23:46:33.518326 4797 raft_consensus.cc:2726] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:46:33.850359 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:33.849878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:58731)
0928 23:46:33.850191 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:33.850260 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260928 23:46:33.974326 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:33.973875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:46493)
0928 23:46:33.974165 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:33.974230 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
I20260928 23:46:34.173198 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:34.172707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:55215)
0928 23:46:34.173003 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:34.173100 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260928 23:46:34.851781 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:34.851346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:34603)
0928 23:46:34.851630 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:34.851685 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":75,"threads_started":1}
I20260928 23:46:34.975636 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:34.975214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:53083)
0928 23:46:34.975464 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:34.975524 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260928 23:46:35.174624 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:35.174163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:42519)
0928 23:46:35.174445 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:35.174521 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260928 23:46:35.853080 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:35.852666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:35333)
0928 23:46:35.852922 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:35.852982 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260928 23:46:35.977326 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:35.976732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:50303)
0928 23:46:35.977080 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:35.977193 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260928 23:46:36.175976 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:36.175521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:54263)
0928 23:46:36.175814 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:36.175880 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260928 23:46:36.647775 4807 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:36.647917 4807 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:36.648557 4807 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177), 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
W20260928 23:46:36.649495 4402 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
W20260928 23:46:36.649713 4400 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:36.649809 4400 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0f121ee55bf4baa92f89b7306dc8c9c; no voters: 4e0439409b474559acdcdc933b4a8a88, 4e75d7000058444ca07fc3bd1540c506
I20260928 23:46:36.649971 4807 raft_consensus.cc:2726] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:46:36.854406 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:36.854002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:47129)
0928 23:46:36.854252 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:36.854311 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260928 23:46:36.978847 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:36.978204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:37125)
0928 23:46:36.978596 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:36.978696 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
I20260928 23:46:37.178020 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:37.177119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:42617)
0928 23:46:37.177489 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:37.177873 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":86,"threads_started":1}
I20260928 23:46:37.855700 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:37.855251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:50155)
0928 23:46:37.855535 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:37.855597 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260928 23:46:37.980356 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:37.979789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:53133)
0928 23:46:37.980186 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:37.980255 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260928 23:46:38.179728 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:38.179275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:47405)
0928 23:46:38.179566 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:38.179632 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260928 23:46:38.856866 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:38.856450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:51069)
0928 23:46:38.856712 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:38.856767 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260928 23:46:38.981707 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:38.981209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:47553)
0928 23:46:38.981491 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:38.981609 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260928 23:46:39.181123 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:39.180576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:46785)
0928 23:46:39.180868 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:39.180944 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260928 23:46:39.309943 4817 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:39.310112 4817 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:39.310562 4817 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177), 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
W20260928 23:46:39.311409 4402 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
W20260928 23:46:39.311502 4400 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:39.311546 4402 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0f121ee55bf4baa92f89b7306dc8c9c; no voters: 4e0439409b474559acdcdc933b4a8a88, 4e75d7000058444ca07fc3bd1540c506
I20260928 23:46:39.311681 4817 raft_consensus.cc:2726] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:46:39.858222 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:39.857772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:55937)
0928 23:46:39.858037 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:39.858093 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260928 23:46:39.982934 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:39.982497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:56623)
0928 23:46:39.982763 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:39.982826 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260928 23:46:40.182590 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:40.182154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:36507)
0928 23:46:40.182426 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:40.182488 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260928 23:46:40.859428 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:40.858962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:40545)
0928 23:46:40.859267 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:40.859326 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260928 23:46:40.984203 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:40.983766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:42459)
0928 23:46:40.984042 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:40.984102 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260928 23:46:41.183840 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:41.183402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:42025)
0928 23:46:41.183681 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:41.183744 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260928 23:46:41.860646 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:41.860209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:60307)
0928 23:46:41.860476 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:41.860540 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260928 23:46:41.985661 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:41.985100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:47033)
0928 23:46:41.985402 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:41.985469 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260928 23:46:42.185123 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:42.184679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:56979)
0928 23:46:42.184955 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:42.185019 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260928 23:46:42.430245 4827 raft_consensus.cc:482] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:42.430409 4827 raft_consensus.cc:504] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e75d7000058444ca07fc3bd1540c506" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 42177 } } peers { permanent_uuid: "4e0439409b474559acdcdc933b4a8a88" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38549 } } peers { permanent_uuid: "d0f121ee55bf4baa92f89b7306dc8c9c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 45163 } }
I20260928 23:46:42.431011 4827 leader_election.cc:288] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177), 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549)
W20260928 23:46:42.431751 4402 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e75d7000058444ca07fc3bd1540c506 (127.21.112.67:42177): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
W20260928 23:46:42.431896 4400 leader_election.cc:341] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e0439409b474559acdcdc933b4a8a88 (127.21.112.66:38549): Not found: Tablet not found: 2c262ab4b58e4f8ca61d7f131438c711
I20260928 23:46:42.432004 4400 leader_election.cc:302] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0f121ee55bf4baa92f89b7306dc8c9c; no voters: 4e0439409b474559acdcdc933b4a8a88, 4e75d7000058444ca07fc3bd1540c506
I20260928 23:46:42.432181 4827 raft_consensus.cc:2726] T 2c262ab4b58e4f8ca61d7f131438c711 P d0f121ee55bf4baa92f89b7306dc8c9c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:46:42.861980 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:42.861499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:55481)
0928 23:46:42.861796 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:42.861852 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":123,"threads_started":1}
I20260928 23:46:42.986925 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:42.986500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:52371)
0928 23:46:42.986769 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:42.986829 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260928 23:46:43.186307 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:43.185894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:33023)
0928 23:46:43.186156 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:43.186214 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260928 23:46:43.863286 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:43.862838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:38073)
0928 23:46:43.863128 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:43.863191 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260928 23:46:43.988197 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:43.987789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:46475)
0928 23:46:43.988043 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:43.988106 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260928 23:46:44.187527 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:44.187077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.67:48917)
0928 23:46:44.187371 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:44.187429 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
I20260928 23:46:44.864830 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:44.864377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.65:34919)
0928 23:46:44.864667 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:44.864734 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260928 23:46:44.989424 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:44.989032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43455 (local address 127.21.112.66:34003)
0928 23:46:44.989278 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:44.989332 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260928 23:46:45.175400 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 4350
I20260928 23:46:45.181340 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 4517
I20260928 23:46:45.187052 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 4651
2026-09-28T23:46:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19491 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:46:45 dist-test-slave-c3s8 krb5kdc[4838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:46:45 dist-test-slave-c3s8 krb5kdc[4838](info): set up 2 sockets
Sep 28 23:46:45 dist-test-slave-c3s8 krb5kdc[4838](info): commencing operation
krb5kdc: starting...
W20260928 23:46:46.246275 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:46:46 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:46:46.373922 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36833
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:46.505788 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:46.506165 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:46.506253 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:46.510444 4851 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:46:46.510528 4851 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:46:46.510556 4851 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:46:46.510581 4851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:46:46.510644 4851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:46.510690 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:46.510731 4851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:46:46.510767 4851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:46:46.516220 4851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:46.517387 4851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:46.518519 4851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:46:46.528189 4851 server_base.cc:1034] running on GCE node
W20260928 23:46:46.528335 4857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:46.528347 4859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:46.528335 4856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:46.529227 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:46.530608 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:46.531844 4851 hybrid_clock.cc:648] HybridClock initialized: now 1790639206531814 us; error 42 us; skew 500 ppm
Sep 28 23:46:46 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639206, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:46.534749 4851 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:46:46.535681 4851 webserver.cc:460] Webserver started at http://127.21.112.126:44267/ using document root <none> and password file <none>
I20260928 23:46:46.536266 4851 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:46.536370 4851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:46.536620 4851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:46.538823 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "5dd58ae18a114aa98974125f8c384192"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "4fc9519eb9fcc7f7d76dd69bd9e32572"
server_key_iv: "ddaf2015a0d6f3d5534d8cb74f9aed24"
server_key_version: "encryptionkey@0"
I20260928 23:46:46.539390 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "5dd58ae18a114aa98974125f8c384192"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "4fc9519eb9fcc7f7d76dd69bd9e32572"
server_key_iv: "ddaf2015a0d6f3d5534d8cb74f9aed24"
server_key_version: "encryptionkey@0"
I20260928 23:46:46.543334 4851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:46:46.545921 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:46.547195 4851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:46:46.547349 4851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "5dd58ae18a114aa98974125f8c384192"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "4fc9519eb9fcc7f7d76dd69bd9e32572"
server_key_iv: "ddaf2015a0d6f3d5534d8cb74f9aed24"
server_key_version: "encryptionkey@0"
I20260928 23:46:46.547467 4851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:46.565748 4851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:46.568969 4851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:46.569201 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:46.576644 4851 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36833
I20260928 23:46:46.576933 4918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36833 every 8 connection(s)
I20260928 23:46:46.577750 4851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:46:46.580653 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4851
I20260928 23:46:46.580684 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:46.580803 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:46:46.581089 21953 external_mini_cluster.cc:1428] Setting key 65e37bb493d6edddfd47fcb1f3c90f58
I20260928 23:46:46.588156 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap starting.
I20260928 23:46:46.590708 4919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Neither blocks nor log segments found. Creating new log.
Sep 28 23:46:46 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:46.591501 4919 log.cc:824] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:46.594105 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: No bootstrap required, opened a new log
I20260928 23:46:46.597162 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:46.582586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.21.112.126:36833)
0928 23:46:46.583048 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:46.583072 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:46.583118 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:46:46.584284 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:46.584311 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:46.584770 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:46.585163 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:46.585926 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:46.587440 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:46.588306 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:46.588955 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:46.589112 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:46.591788 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:46.591794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:46.591810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:46.591842 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:46.593871 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:46.594409 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:46.594415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:46.594421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:46.594573 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:46.594834 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:46.594837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:46.594839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:46.595351 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:46.595536 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:46.595795 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 23:46:46.596099 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":130,"threads_started":1}
I20260928 23:46:46.598587 4919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:46:46.598824 4919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:46.598892 4919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd58ae18a114aa98974125f8c384192, State: Initialized, Role: FOLLOWER
I20260928 23:46:46.599334 4919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:46:46.599504 4919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:46:46.599602 4919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:46:46.599716 4919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:46.600692 4919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:46:46.601052 4919 leader_election.cc:302] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dd58ae18a114aa98974125f8c384192; no voters:
I20260928 23:46:46.601401 4919 leader_election.cc:288] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:46:46.601519 4924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:46.601825 4924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd58ae18a114aa98974125f8c384192, State: Running, Role: LEADER
I20260928 23:46:46.602192 4924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:46:46.602720 4919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:46:46.603960 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd58ae18a114aa98974125f8c384192. Latest consensus state: current_term: 1 leader_uuid: "5dd58ae18a114aa98974125f8c384192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } } }
I20260928 23:46:46.604089 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: This master's current role is: LEADER
I20260928 23:46:46.604504 4925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dd58ae18a114aa98974125f8c384192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } } }
I20260928 23:46:46.605141 4925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: This master's current role is: LEADER
W20260928 23:46:46.607643 4939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:46:46.607736 4939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:46:46.607817 4933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:46:46.608538 4933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:46:46.614552 4933 catalog_manager.cc:1347] Generated new cluster ID: c181876993564f7c83d2ab89a2d16c35
I20260928 23:46:46.614635 4933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:46:46.649365 4933 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:46:46.650607 4933 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:46:46.656821 4933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Generated new TSK 0
I20260928 23:46:46.657760 4933 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:46:46.689869 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36833
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:46.810393 4945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:46.810719 4945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:46.810815 4945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:46.814994 4945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:46.815092 4945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:46.815189 4945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:46:46.820806 4945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:46.821997 4945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:46.823061 4945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:46.833281 4951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:46.833295 4950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:46.833323 4953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:46.833645 4945 server_base.cc:1034] running on GCE node
I20260928 23:46:46.834161 4945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:46.834822 4945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:46.835996 4945 hybrid_clock.cc:648] HybridClock initialized: now 1790639206835956 us; error 51 us; skew 500 ppm
Sep 28 23:46:46 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639206, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:46.838923 4945 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:46:46.839859 4945 webserver.cc:460] Webserver started at http://127.21.112.65:45187/ using document root <none> and password file <none>
I20260928 23:46:46.840448 4945 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:46.840540 4945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:46.840785 4945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:46.842907 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "d03106721b4645abbc58c5d949a05dc5"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "c8f3e91417810a0b5f190818eb066c6a"
server_key_iv: "5b1e92444aa082730d016b3cabff04f4"
server_key_version: "encryptionkey@0"
I20260928 23:46:46.843492 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "d03106721b4645abbc58c5d949a05dc5"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "c8f3e91417810a0b5f190818eb066c6a"
server_key_iv: "5b1e92444aa082730d016b3cabff04f4"
server_key_version: "encryptionkey@0"
I20260928 23:46:46.847419 4945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:46:46.849906 4960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:46.851274 4945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:46:46.851418 4945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "d03106721b4645abbc58c5d949a05dc5"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "c8f3e91417810a0b5f190818eb066c6a"
server_key_iv: "5b1e92444aa082730d016b3cabff04f4"
server_key_version: "encryptionkey@0"
I20260928 23:46:46.851536 4945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:46.874040 4945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:46.877110 4945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:46.877329 4945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:46.878005 4945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:46.878886 4945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:46.878957 4945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:46.879029 4945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:46.879063 4945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:46.888077 4945 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:42207
I20260928 23:46:46.888134 5073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:42207 every 8 connection(s)
I20260928 23:46:46.889042 4945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:46:46.897615 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 4945
I20260928 23:46:46.897759 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:46:46.898136 21953 external_mini_cluster.cc:1428] Setting key e2d9c33e3dab202175332232c12c4640
Sep 28 23:46:46 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639206, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:46.904754 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:46.890959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:54127 (local address 127.21.112.126:36833)
0928 23:46:46.891118 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:46.891123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:46.891989 (+ 866us) server_negotiation.cc:407] Connection header received
0928 23:46:46.893328 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:46.893331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:46.893381 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:46.893458 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:46.894787 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:46.895356 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:46.896553 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:46.896802 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:46.896894 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:46.900022 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:46.900026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:46.900028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:46.900053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:46.901966 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:46.902624 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:46.902629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:46.902634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:46.902733 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:46.903231 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:46.903234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:46.903235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:46.903491 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:46.903595 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:46.904444 (+ 849us) server_negotiation.cc:299] Negotiation successful
0928 23:46:46.904557 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:46:46.906155 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:46.891339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:54127)
0928 23:46:46.891825 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:46.891870 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:46.893114 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:46.893717 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:46.893723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:46.894113 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:46.894620 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:46.894637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:46.895479 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:46.895482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:46.896404 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:46.896413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:46.896952 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:46.896955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:46.897116 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:46.898050 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:46.898104 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:46.899836 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:46.902135 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:46.902141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:46.902156 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:46.902519 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:46.902890 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:46.902896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:46.902901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:46.903127 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:46.903599 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:46.903609 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:46.904014 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 23:46:46.904395 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 23:46:46.904831 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":120,"threads_started":1}
I20260928 23:46:46.907552 5074 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36833
I20260928 23:46:46.907830 5074 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:46.908488 5074 heartbeater.cc:511] Master 127.21.112.126:36833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:46:46.910239 4883 ts_manager.cc:194] Registered new tserver with Master: d03106721b4645abbc58c5d949a05dc5 (127.21.112.65:42207)
I20260928 23:46:46.911473 4883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:54127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:46:46.929853 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36833
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:47.058264 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:47.058588 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:47.058696 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:47.062885 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:47.062983 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:47.063074 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:46:47.068647 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:47.069931 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:47.070994 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:47.080999 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:47.081027 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:47.081059 5079 server_base.cc:1034] running on GCE node
W20260928 23:46:47.080998 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:47.081768 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:47.082460 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:47.083621 5079 hybrid_clock.cc:648] HybridClock initialized: now 1790639207083588 us; error 48 us; skew 500 ppm
Sep 28 23:46:47 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:47.086494 5079 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:46:47.087378 5079 webserver.cc:460] Webserver started at http://127.21.112.66:45463/ using document root <none> and password file <none>
I20260928 23:46:47.087935 5079 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:47.087997 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:47.088196 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:47.090425 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "09865c7b6d1d413386b075b5ed927616"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "a82bf697d728f54339121c2d31e36040"
server_key_iv: "1a36944ea5ae0a6033b7dbc8310df546"
server_key_version: "encryptionkey@0"
I20260928 23:46:47.090963 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "09865c7b6d1d413386b075b5ed927616"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "a82bf697d728f54339121c2d31e36040"
server_key_iv: "1a36944ea5ae0a6033b7dbc8310df546"
server_key_version: "encryptionkey@0"
I20260928 23:46:47.094770 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 23:46:47.097507 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:47.099074 5079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:46:47.099182 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "09865c7b6d1d413386b075b5ed927616"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "a82bf697d728f54339121c2d31e36040"
server_key_iv: "1a36944ea5ae0a6033b7dbc8310df546"
server_key_version: "encryptionkey@0"
I20260928 23:46:47.099282 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:47.113148 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:47.116658 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:47.116822 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:47.117386 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:47.118259 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:47.118307 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:47.118346 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:47.118368 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:47.127434 5079 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:42429
I20260928 23:46:47.127454 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:42429 every 8 connection(s)
I20260928 23:46:47.128487 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:46:47.136499 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5079
I20260928 23:46:47.136628 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:46:47.136965 21953 external_mini_cluster.cc:1428] Setting key 8201dcbdfd02df69133836071bc94a6a
Sep 28 23:46:47 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:47.143867 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.130652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:56243 (local address 127.21.112.126:36833)
0928 23:46:47.130804 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.130808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.131638 (+ 830us) server_negotiation.cc:407] Connection header received
0928 23:46:47.133035 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.133039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.133099 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.133217 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:47.134498 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.135026 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.136188 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.136571 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.136689 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.139730 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:47.139737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:47.139743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:47.139777 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:47.141487 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:47.142032 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:47.142039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:47.142043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:47.142143 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:47.142482 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:47.142489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:47.142493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:47.142765 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:47.142881 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.143465 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.143627 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:46:47.144596 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.130989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:56243)
0928 23:46:47.131488 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.131532 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.132815 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.133393 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.133401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.133841 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:47.134350 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.134364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.135159 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.135167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.136049 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.136058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.136772 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.136775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.136980 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.137702 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:47.137719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:47.139545 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:47.141641 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:47.141646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:47.141660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:47.141927 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:47.142248 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:47.142251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:47.142252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:47.142389 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:47.142897 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:47.142905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:47.143201 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.143374 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.143651 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":157,"threads_started":1}
I20260928 23:46:47.145955 5208 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36833
I20260928 23:46:47.146210 5208 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:47.146756 5208 heartbeater.cc:511] Master 127.21.112.126:36833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:46:47.147956 4883 ts_manager.cc:194] Registered new tserver with Master: 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
I20260928 23:46:47.148546 4883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:56243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:46:47.169982 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36833
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:46:47.289170 5213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:46:47.289443 5213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:46:47.289486 5213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:46:47.293685 5213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:46:47.293746 5213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:46:47.293850 5213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:46:47.299238 5213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:46:47.300485 5213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:46:47.301592 5213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:46:47.312249 5219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:46:47.312268 5218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:47.312428 5213 server_base.cc:1034] running on GCE node
W20260928 23:46:47.312330 5221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:46:47.313021 5213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:46:47.313777 5213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:46:47.314970 5213 hybrid_clock.cc:648] HybridClock initialized: now 1790639207314915 us; error 65 us; skew 500 ppm
Sep 28 23:46:47 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:46:47.317870 5213 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:46:47.318814 5213 webserver.cc:460] Webserver started at http://127.21.112.67:44717/ using document root <none> and password file <none>
I20260928 23:46:47.319406 5213 fs_manager.cc:359] Metadata directory not provided
I20260928 23:46:47.319499 5213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:46:47.319736 5213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:46:47.321861 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "c6d84b054ad7424583f17e2831a7b3c0"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "c154e662c0962b90ab31bca0858b480e"
server_key_iv: "13f35f9eed15643ee4679e1d509595d4"
server_key_version: "encryptionkey@0"
I20260928 23:46:47.322433 5213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "c6d84b054ad7424583f17e2831a7b3c0"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "c154e662c0962b90ab31bca0858b480e"
server_key_iv: "13f35f9eed15643ee4679e1d509595d4"
server_key_version: "encryptionkey@0"
I20260928 23:46:47.326335 5213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 23:46:47.328722 5228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:47.330078 5213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 23:46:47.330221 5213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "c6d84b054ad7424583f17e2831a7b3c0"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "c154e662c0962b90ab31bca0858b480e"
server_key_iv: "13f35f9eed15643ee4679e1d509595d4"
server_key_version: "encryptionkey@0"
I20260928 23:46:47.330345 5213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:46:47.347083 5213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:46:47.350623 5213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:46:47.350821 5213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:46:47.351532 5213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:46:47.352433 5213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:46:47.352480 5213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:47.352541 5213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:46:47.352574 5213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:46:47.361662 5213 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:45473
I20260928 23:46:47.361681 5341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:45473 every 8 connection(s)
I20260928 23:46:47.362601 5213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:46:47.366717 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5213
I20260928 23:46:47.366864 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:46:47.367200 21953 external_mini_cluster.cc:1428] Setting key eb7ecc48eabc01ba811b968aafa16224
Sep 28 23:46:47 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:46:47.376929 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.364503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:34171 (local address 127.21.112.126:36833)
0928 23:46:47.364646 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.364650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.365460 (+ 810us) server_negotiation.cc:407] Connection header received
0928 23:46:47.366933 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.366937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.366998 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.367109 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:47.368470 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.369198 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.370516 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.370754 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.370866 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.373288 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:47.373292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:47.373295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:47.373327 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:47.374843 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:47.375384 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:47.375388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:47.375390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:47.375474 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:47.375827 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:47.375831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:47.375833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:47.376020 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:47.376124 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.376639 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.376768 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:46:47.377821 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.364884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:34171)
0928 23:46:47.365320 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.365362 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.366696 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.367287 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.367298 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.367716 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:46:47.368331 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.368347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.369333 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.369336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.370387 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.370397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.370904 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.370908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.371089 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.371700 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:46:47.371716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:46:47.373117 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:46:47.374981 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:47.374987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:47.375003 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:47.375274 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:47.375566 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:46:47.375570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:46:47.375572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:46:47.375727 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:46:47.376134 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:46:47.376143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:46:47.376394 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.376590 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.376933 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260928 23:46:47.379117 5342 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36833
I20260928 23:46:47.379364 5342 heartbeater.cc:464] Registering TS with master...
I20260928 23:46:47.379875 5342 heartbeater.cc:511] Master 127.21.112.126:36833 requested a full tablet report, sending...
I20260928 23:46:47.380937 4883 ts_manager.cc:194] Registered new tserver with Master: c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473)
I20260928 23:46:47.381575 4883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:34171
I20260928 23:46:47.393199 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:46:47.402710 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.395202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.21.112.126:36833)
0928 23:46:47.395358 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.395364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.395541 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:46:47.395706 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.395712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.395757 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.395837 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:46:47.396556 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.397082 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.398019 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.398243 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.398351 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.399760 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:46:47.399763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:46:47.399766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:46:47.399796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:46:47.400967 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:47.401389 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:47.401394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:47.401396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:47.401477 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:46:47.401785 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:46:47.401788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:46:47.401790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:46:47.402014 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:46:47.402152 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.402401 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.402548 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:46:47.405756 4883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:46:47.408756 4883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:46:47.424391 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.417800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.21.112.66:42429)
0928 23:46:47.418517 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.418531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.419070 (+ 539us) server_negotiation.cc:407] Connection header received
0928 23:46:47.419338 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.419342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.419549 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.419677 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:47.420258 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.421388 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.422754 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.423579 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.423702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.423885 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.424084 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.424213 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":134,"threads_started":1}
I20260928 23:46:47.424721 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.418183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:42207 (local address 127.0.0.1:56172)
0928 23:46:47.419139 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.419152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.419284 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.419920 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.419924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.419954 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:47.420119 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.420124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.421303 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.421306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.423066 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.423077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.424235 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.424239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.424333 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.424355 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.424422 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.424572 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":166,"spinlock_wait_cycles":896,"thread_start_us":106,"threads_started":1}
I20260928 23:46:47.424999 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.418510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.21.112.67:45473)
0928 23:46:47.418946 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.418978 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.419259 (+ 281us) server_negotiation.cc:407] Connection header received
0928 23:46:47.419336 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.419341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.419549 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.419698 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:47.420062 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.421266 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.423214 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.423845 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.423945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.424093 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.424699 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.424818 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":112,"threads_started":1}
I20260928 23:46:47.425148 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.417716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42429 (local address 127.0.0.1:37596)
0928 23:46:47.418928 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.418982 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.419126 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.419890 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.419892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.419907 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:47.420111 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.420121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.421553 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.421557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.422610 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.422621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.423728 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.423732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.423845 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.423900 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.424925 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.425004 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":171,"spinlock_wait_cycles":896,"thread_start_us":77,"threads_started":1}
I20260928 23:46:47.425392 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.417847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45473 (local address 127.0.0.1:43270)
0928 23:46:47.418928 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.418982 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.419116 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.419719 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.419727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.419755 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:47.419910 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.419918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.421423 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.421428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.423070 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.423077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.423990 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.423994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.424097 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.424123 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.425167 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.425264 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":209,"thread_start_us":138,"threads_started":1}
I20260928 23:46:47.425390 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.418618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.21.112.65:42207)
0928 23:46:47.419072 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.419079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.419251 (+ 172us) server_negotiation.cc:407] Connection header received
0928 23:46:47.419442 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.419447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.419612 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.419784 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:47.420234 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.421170 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.423204 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.424106 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.424232 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.424381 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.424463 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.424568 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":111,"threads_started":1}
I20260928 23:46:47.428226 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 211897546e304d8384566ace9f2591dc (DEFAULT_TABLE table=test-table [id=4ba661d54b2b4ad8b41b2ae24feaa842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:47.428176 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 211897546e304d8384566ace9f2591dc (DEFAULT_TABLE table=test-table [id=4ba661d54b2b4ad8b41b2ae24feaa842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:47.428223 5008 tablet_service.cc:1467] Processing CreateTablet for tablet 211897546e304d8384566ace9f2591dc (DEFAULT_TABLE table=test-table [id=4ba661d54b2b4ad8b41b2ae24feaa842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:46:47.429339 5008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211897546e304d8384566ace9f2591dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:47.429339 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211897546e304d8384566ace9f2591dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:47.429353 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211897546e304d8384566ace9f2591dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:46:47.436326 5359 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Bootstrap starting.
I20260928 23:46:47.437072 5360 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Bootstrap starting.
I20260928 23:46:47.437508 5361 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Bootstrap starting.
I20260928 23:46:47.440213 5359 tablet_bootstrap.cc:654] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:47.440264 5361 tablet_bootstrap.cc:654] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:47.441267 5361 log.cc:824] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:47.441601 5359 log.cc:824] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:47.443351 5361 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: No bootstrap required, opened a new log
I20260928 23:46:47.443594 5361 ts_tablet_manager.cc:1397] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 23:46:47.443650 5359 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: No bootstrap required, opened a new log
I20260928 23:46:47.443648 5360 tablet_bootstrap.cc:654] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Neither blocks nor log segments found. Creating new log.
I20260928 23:46:47.443857 5359 ts_tablet_manager.cc:1397] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 23:46:47.444468 5360 log.cc:824] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:46:47.446743 5360 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: No bootstrap required, opened a new log
I20260928 23:46:47.446977 5360 ts_tablet_manager.cc:1397] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260928 23:46:47.447619 5361 raft_consensus.cc:348] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.447628 5359 raft_consensus.cc:348] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.447887 5361 raft_consensus.cc:374] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:47.447897 5359 raft_consensus.cc:374] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:47.447957 5359 raft_consensus.cc:729] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d84b054ad7424583f17e2831a7b3c0, State: Initialized, Role: FOLLOWER
I20260928 23:46:47.447970 5361 raft_consensus.cc:729] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09865c7b6d1d413386b075b5ed927616, State: Initialized, Role: FOLLOWER
I20260928 23:46:47.448460 5359 consensus_queue.cc:260] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.448535 5361 consensus_queue.cc:260] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.449255 5342 heartbeater.cc:503] Master 127.21.112.126:36833 was elected leader, sending a full tablet report...
I20260928 23:46:47.449628 5359 ts_tablet_manager.cc:1428] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:46:47.449645 5361 ts_tablet_manager.cc:1428] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 23:46:47.449996 5208 heartbeater.cc:503] Master 127.21.112.126:36833 was elected leader, sending a full tablet report...
I20260928 23:46:47.451516 5360 raft_consensus.cc:348] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.451820 5360 raft_consensus.cc:374] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:46:47.451891 5360 raft_consensus.cc:729] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03106721b4645abbc58c5d949a05dc5, State: Initialized, Role: FOLLOWER
I20260928 23:46:47.452374 5360 consensus_queue.cc:260] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.453032 5074 heartbeater.cc:503] Master 127.21.112.126:36833 was elected leader, sending a full tablet report...
I20260928 23:46:47.453325 5360 ts_tablet_manager.cc:1428] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:46:47.471282 5367 raft_consensus.cc:482] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:46:47.471496 5367 raft_consensus.cc:504] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.472615 5367 leader_election.cc:288] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429), c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473)
I20260928 23:46:47.476663 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.472987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:40033 (local address 127.21.112.67:45473)
0928 23:46:47.473124 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.473130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.473379 (+ 249us) server_negotiation.cc:407] Connection header received
0928 23:46:47.473568 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.473571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.473624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.473708 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:47.474010 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.474536 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.475686 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.476082 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.476215 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.476267 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.476459 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.476531 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:46:47.476683 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.472972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45061 (local address 127.21.112.66:42429)
0928 23:46:47.473114 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.473118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.473480 (+ 362us) server_negotiation.cc:407] Connection header received
0928 23:46:47.473689 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.473694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.473748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.473833 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:46:47.474129 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.474618 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.475696 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.476096 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.476240 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.476298 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.476487 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.476557 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 23:46:47.476722 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.472868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45473 (local address 127.21.112.65:40033)
0928 23:46:47.473279 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.473291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.473425 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.473722 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.473725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.473740 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:47.473891 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.473896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.474654 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.474658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.475546 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.475555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.476212 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.476215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.476311 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.476333 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.476444 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.476517 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":29,"thread_start_us":138,"threads_started":1}
I20260928 23:46:47.476733 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.472868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42429 (local address 127.21.112.65:45061)
0928 23:46:47.473385 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.473397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:46:47.473518 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:46:47.473845 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:46:47.473848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:46:47.473863 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:46:47.474012 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.474017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.474736 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.474741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.475571 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:46:47.475580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.476239 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.476242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:46:47.476357 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.476382 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:46:47.476471 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:46:47.476541 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":86,"thread_start_us":115,"threads_started":1}
I20260928 23:46:47.477512 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211897546e304d8384566ace9f2591dc" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d84b054ad7424583f17e2831a7b3c0" is_pre_election: true
I20260928 23:46:47.477835 5296 raft_consensus.cc:2454] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 0.
I20260928 23:46:47.477893 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211897546e304d8384566ace9f2591dc" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09865c7b6d1d413386b075b5ed927616" is_pre_election: true
I20260928 23:46:47.478178 5162 raft_consensus.cc:2454] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 0.
I20260928 23:46:47.478434 4963 leader_election.cc:302] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d84b054ad7424583f17e2831a7b3c0, d03106721b4645abbc58c5d949a05dc5; no voters:
I20260928 23:46:47.478757 5367 raft_consensus.cc:2781] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:46:47.478844 5367 raft_consensus.cc:482] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:46:47.478888 5367 raft_consensus.cc:3037] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:47.479883 5367 raft_consensus.cc:504] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.480219 5367 leader_election.cc:288] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 election: Requested vote from peers 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429), c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473)
I20260928 23:46:47.480557 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211897546e304d8384566ace9f2591dc" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d84b054ad7424583f17e2831a7b3c0"
I20260928 23:46:47.480571 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211897546e304d8384566ace9f2591dc" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09865c7b6d1d413386b075b5ed927616"
I20260928 23:46:47.480667 5296 raft_consensus.cc:3037] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:47.480700 5162 raft_consensus.cc:3037] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:46:47.481774 5296 raft_consensus.cc:2454] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 1.
I20260928 23:46:47.481851 5162 raft_consensus.cc:2454] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 1.
I20260928 23:46:47.482076 4963 leader_election.cc:302] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d84b054ad7424583f17e2831a7b3c0, d03106721b4645abbc58c5d949a05dc5; no voters:
I20260928 23:46:47.482263 5367 raft_consensus.cc:2781] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:46:47.482512 5367 raft_consensus.cc:686] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 LEADER]: Becoming Leader. State: Replica: d03106721b4645abbc58c5d949a05dc5, State: Running, Role: LEADER
I20260928 23:46:47.482851 5367 consensus_queue.cc:237] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:46:47.486227 4883 catalog_manager.cc:5495] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d03106721b4645abbc58c5d949a05dc5 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "d03106721b4645abbc58c5d949a05dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } health_report { overall_health: HEALTHY } } }
I20260928 23:46:47.502928 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.499172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.21.112.65:42207)
0928 23:46:47.499317 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:46:47.499321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:46:47.499335 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:46:47.499490 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:46:47.499492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:46:47.499546 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:46:47.499652 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:46:47.499968 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.500486 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.501342 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:46:47.501625 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:46:47.501708 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:46:47.502023 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:46:47.502481 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:46:47.502588 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:46:47.502690 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 23:46:47.502742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:46:47.509384 5162 raft_consensus.cc:1261] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Refusing update from remote peer d03106721b4645abbc58c5d949a05dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:47.509981 5296 raft_consensus.cc:1261] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Refusing update from remote peer d03106721b4645abbc58c5d949a05dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:46:47.510134 5370 consensus_queue.cc:1035] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:47.510610 5367 consensus_queue.cc:1035] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:46:47.518790 5375 mvcc.cc:205] Tried to move back new op lower bound from 7334458193950498816 to 7334458193852362752. Current Snapshot: MvccSnapshot[applied={T|T < 7334458193950498816}]
I20260928 23:46:47.519263 5374 mvcc.cc:205] Tried to move back new op lower bound from 7334458193950498816 to 7334458193852362752. Current Snapshot: MvccSnapshot[applied={T|T < 7334458193950498816}]
I20260928 23:46:47.527637 4883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:47.527843 4883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:46:47.530725 4883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Sending DeleteTablet for 3 replicas of tablet 211897546e304d8384566ace9f2591dc
I20260928 23:46:47.531607 5008 tablet_service.cc:1514] Processing DeleteTablet for tablet 211897546e304d8384566ace9f2591dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:47 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:56172
I20260928 23:46:47.531607 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 211897546e304d8384566ace9f2591dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:47 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37596
I20260928 23:46:47.531827 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 4945
I20260928 23:46:47.533247 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 211897546e304d8384566ace9f2591dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:47 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:43270
I20260928 23:46:47.533468 5382 tablet_replica.cc:330] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: stopping tablet replica
I20260928 23:46:47.533828 5383 tablet_replica.cc:330] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: stopping tablet replica
I20260928 23:46:47.533865 5382 raft_consensus.cc:2229] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:47.534284 5382 raft_consensus.cc:2258] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:47.534224 5383 raft_consensus.cc:2229] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:46:47.534595 5383 raft_consensus.cc:2258] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:46:47.535763 5382 ts_tablet_manager.cc:1905] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:47.536042 5383 ts_tablet_manager.cc:1905] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:46:47.538196 5382 ts_tablet_manager.cc:1918] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:47.538307 5382 log.cc:1196] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/211897546e304d8384566ace9f2591dc
I20260928 23:46:47.538591 5382 ts_tablet_manager.cc:1939] T 211897546e304d8384566ace9f2591dc P 09865c7b6d1d413386b075b5ed927616: Deleting consensus metadata
I20260928 23:46:47.539405 5383 ts_tablet_manager.cc:1918] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:46:47.539515 5383 log.cc:1196] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/211897546e304d8384566ace9f2591dc
I20260928 23:46:47.539834 5383 ts_tablet_manager.cc:1939] T 211897546e304d8384566ace9f2591dc P c6d84b054ad7424583f17e2831a7b3c0: Deleting consensus metadata
I20260928 23:46:47.540112 4870 catalog_manager.cc:4841] TS 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429): tablet 211897546e304d8384566ace9f2591dc (table test-table [id=4ba661d54b2b4ad8b41b2ae24feaa842]) successfully deleted
W20260928 23:46:47.540556 4867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:42207 (error 108)
I20260928 23:46:47.541002 4869 catalog_manager.cc:4841] TS c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473): tablet 211897546e304d8384566ace9f2591dc (table test-table [id=4ba661d54b2b4ad8b41b2ae24feaa842]) successfully deleted
I20260928 23:46:47.541397 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5079
I20260928 23:46:47.541759 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:46:47.541404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:42207 (local address 127.0.0.1:56186)
0928 23:46:47.541572 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 23:46:47.541658 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:42207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 23:46:47.542148 4867 catalog_manager.cc:4584] TS d03106721b4645abbc58c5d949a05dc5 (127.21.112.65:42207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 211897546e304d8384566ace9f2591dc: Network error: Client connection negotiation failed: client connection to 127.21.112.65:42207: connect: Connection refused (error 111)
I20260928 23:46:47.547432 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5213
I20260928 23:46:47.553313 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 4851
I20260928 23:47:07.560071 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36833
--webserver_interface=127.21.112.126
--webserver_port=44267
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:07.676087 5385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:07.676394 5385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:07.676488 5385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:07.680573 5385 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:47:07.680658 5385 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:47:07.680687 5385 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:47:07.680720 5385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:07.680764 5385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:07.680817 5385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:07.680855 5385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:07.680891 5385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:07.686275 5385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=44267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:07.687414 5385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:07.688503 5385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:07.703055 5394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:07.703039 5391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:07.703063 5392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:07.703220 5385 server_base.cc:1034] running on GCE node
I20260928 23:47:07.704056 5385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:07.705302 5385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:07.706619 5385 hybrid_clock.cc:648] HybridClock initialized: now 1790639227706580 us; error 47 us; skew 500 ppm
Sep 28 23:47:07 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:07.709694 5385 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:47:07.710667 5385 webserver.cc:460] Webserver started at http://127.21.112.126:44267/ using document root <none> and password file <none>
I20260928 23:47:07.711320 5385 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:07.711418 5385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:07.715619 5385 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:47:07.718196 5401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:07.719571 5385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:07.719738 5385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "5dd58ae18a114aa98974125f8c384192"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "4fc9519eb9fcc7f7d76dd69bd9e32572"
server_key_iv: "ddaf2015a0d6f3d5534d8cb74f9aed24"
server_key_version: "encryptionkey@0"
I20260928 23:47:07.720378 5385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:07.733731 5385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:07.736843 5385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:07.737058 5385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:07.745000 5385 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36833
I20260928 23:47:07.745337 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36833 every 8 connection(s)
I20260928 23:47:07.746229 5385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:07.747273 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5385
I20260928 23:47:07.747784 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:42207
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:36833
--webserver_port=45187
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:47:07.751900 5454 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:47:07.753877 5454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap starting.
I20260928 23:47:07.771973 5454 log.cc:824] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:07.780714 5454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:47:07.781100 5454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap complete.
I20260928 23:47:07.785223 5454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:07.785742 5454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd58ae18a114aa98974125f8c384192, State: Initialized, Role: FOLLOWER
I20260928 23:47:07.786269 5454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:07.786409 5454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:07.786501 5454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:07.786600 5454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:47:07.788508 5454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:07.788851 5454 leader_election.cc:302] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dd58ae18a114aa98974125f8c384192; no voters:
I20260928 23:47:07.789212 5454 leader_election.cc:288] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:47:07.789279 5458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:47:07.789525 5458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 LEADER]: Becoming Leader. State: Replica: 5dd58ae18a114aa98974125f8c384192, State: Running, Role: LEADER
I20260928 23:47:07.789908 5458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:07.790494 5454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:07.791384 5459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5dd58ae18a114aa98974125f8c384192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } } }
I20260928 23:47:07.791440 5460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd58ae18a114aa98974125f8c384192. Latest consensus state: current_term: 2 leader_uuid: "5dd58ae18a114aa98974125f8c384192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } } }
I20260928 23:47:07.791534 5459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:07.791556 5460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:07.791951 5467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:07.796999 5467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:07.797765 5467 catalog_manager.cc:1259] Loaded cluster ID: c181876993564f7c83d2ab89a2d16c35
I20260928 23:47:07.797840 5467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:07.799633 5467 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:07.800233 5467 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Loaded TSK: 0
I20260928 23:47:07.820158 5467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Generated new TSK 1
I20260928 23:47:07.820454 5467 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:47:07.890141 5456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:07.890462 5456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:07.890547 5456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:07.895314 5456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:07.895399 5456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:07.895493 5456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:07.900877 5456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:42207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=45187
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:07.902070 5456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:07.903134 5456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:07.913033 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:07.913023 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:07.913026 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:07.913367 5456 server_base.cc:1034] running on GCE node
I20260928 23:47:07.913800 5456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:07.914499 5456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:07.915692 5456 hybrid_clock.cc:648] HybridClock initialized: now 1790639227915650 us; error 49 us; skew 500 ppm
Sep 28 23:47:07 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:07.918479 5456 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:47:07.919425 5456 webserver.cc:460] Webserver started at http://127.21.112.65:45187/ using document root <none> and password file <none>
I20260928 23:47:07.920006 5456 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:07.920099 5456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:07.924209 5456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 23:47:07.926573 5491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:07.928009 5456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:07.928148 5456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "d03106721b4645abbc58c5d949a05dc5"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "c8f3e91417810a0b5f190818eb066c6a"
server_key_iv: "5b1e92444aa082730d016b3cabff04f4"
server_key_version: "encryptionkey@0"
I20260928 23:47:07.928711 5456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:07.945765 5456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:07.949165 5456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:07.949389 5456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:07.950119 5456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:07.951247 5498 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:47:07.956822 5456 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:47:07.956918 5456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 23:47:07.957006 5456 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:47:07.958902 5456 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:47:07.958973 5456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 23:47:07.959297 5498 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Bootstrap starting.
I20260928 23:47:07.969640 5456 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:42207
I20260928 23:47:07.969823 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:42207 every 8 connection(s)
I20260928 23:47:07.970759 5456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:07.975402 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5456
I20260928 23:47:07.975909 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:42429
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:36833
--webserver_port=45463
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:47:07.983580 5498 log.cc:824] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:47:07 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:07.991752 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:07.972813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:53917)
0928 23:47:07.973586 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:07.973642 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:07.974484 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:07.976153 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:07.976164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:07.976712 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:07.977428 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:07.977455 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:07.979541 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:07.979544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:07.980438 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:07.980449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:07.981877 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:07.981884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:07.982165 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:07.982897 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:07.982927 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:07.984962 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:07.987923 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:07.987929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:07.987944 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:07.988322 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:07.988815 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:07.988818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:07.988820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:07.988994 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:07.989846 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:07.989852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:07.990156 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 23:47:07.990367 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 23:47:07.990750 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":126,"threads_started":1}
I20260928 23:47:07.992106 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:07.972774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:53917 (local address 127.21.112.126:36833)
0928 23:47:07.973272 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:07.973294 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:07.973751 (+ 457us) server_negotiation.cc:407] Connection header received
0928 23:47:07.974894 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:07.974903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:07.975263 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:07.975610 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:07.977730 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:07.979371 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:07.980586 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:07.981391 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:07.981617 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:07.985239 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:07.985248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:07.985269 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:07.985313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:07.987757 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:07.988492 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:07.988499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:07.988506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:07.988680 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:07.989115 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:07.989119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:07.989121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:07.989636 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:07.989769 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:07.990554 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 23:47:07.990956 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":112,"threads_started":1}
I20260928 23:47:07.993386 5498 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:47:07.993954 5498 tablet_bootstrap.cc:492] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Bootstrap complete.
I20260928 23:47:07.995285 5606 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36833
I20260928 23:47:07.995378 5498 ts_tablet_manager.cc:1397] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.013s
I20260928 23:47:07.995544 5606 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:07.996135 5606 heartbeater.cc:511] Master 127.21.112.126:36833 requested a full tablet report, sending...
I20260928 23:47:07.998310 5498 raft_consensus.cc:348] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:07.998756 5498 raft_consensus.cc:729] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03106721b4645abbc58c5d949a05dc5, State: Initialized, Role: FOLLOWER
I20260928 23:47:07.999114 5415 ts_manager.cc:194] Registered new tserver with Master: d03106721b4645abbc58c5d949a05dc5 (127.21.112.65:42207)
I20260928 23:47:07.999281 5498 consensus_queue.cc:260] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.000252 5498 ts_tablet_manager.cc:1428] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:47:08.003229 5415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:53917
I20260928 23:47:08.005910 5606 heartbeater.cc:503] Master 127.21.112.126:36833 was elected leader, sending a full tablet report...
Sep 28 23:47:08 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.65@KRBTEST.COM
I20260928 23:47:08.018239 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.002805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.21.112.65:42207)
0928 23:47:08.003171 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.003183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.003613 (+ 430us) server_negotiation.cc:407] Connection header received
0928 23:47:08.003825 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.003836 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.003993 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.004173 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:08.005063 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.006217 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.007529 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.008044 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.008150 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.013745 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:08.013762 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:08.013789 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:08.013826 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:08.015856 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.016612 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.016618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.016619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.016707 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.017042 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.017044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.017046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.017368 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:08.017472 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.017905 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.018057 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":83,"threads_started":1}
I20260928 23:47:08.018270 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.002653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:42207 (local address 127.0.0.1:51280)
0928 23:47:08.003464 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.003513 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.003683 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.004330 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.004355 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.004801 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:08.004930 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.004938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.006337 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.006343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.007295 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.007306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.008176 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.008180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.008295 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.008884 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:08.008896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:08.013462 (+ 4566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:08.016001 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:08.016014 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:08.016020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:08.016454 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:08.016820 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:08.016823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:08.016825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:08.016938 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:08.017485 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:08.017492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:08.017667 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.017833 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.018063 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":123,"threads_started":1}
I20260928 23:47:08.019392 5540 tablet_service.cc:1514] Processing DeleteTablet for tablet 211897546e304d8384566ace9f2591dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:46:47 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51280
I20260928 23:47:08.020388 5615 tablet_replica.cc:330] stopping tablet replica
I20260928 23:47:08.020699 5615 raft_consensus.cc:2229] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:08.020952 5615 raft_consensus.cc:2258] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:08.021970 5615 ts_tablet_manager.cc:1905] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:08.025739 5615 ts_tablet_manager.cc:1918] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:08.025851 5615 log.cc:1196] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal/wals/211897546e304d8384566ace9f2591dc
I20260928 23:47:08.026114 5615 ts_tablet_manager.cc:1939] T 211897546e304d8384566ace9f2591dc P d03106721b4645abbc58c5d949a05dc5: Deleting consensus metadata
I20260928 23:47:08.027882 5402 catalog_manager.cc:4841] TS d03106721b4645abbc58c5d949a05dc5 (127.21.112.65:42207): tablet 211897546e304d8384566ace9f2591dc (table test-table [id=4ba661d54b2b4ad8b41b2ae24feaa842]) successfully deleted
W20260928 23:47:08.103891 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:08.104174 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:08.104228 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:08.108412 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:08.108484 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:08.108547 5610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:08.114121 5610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:42429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=45463
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:08.115249 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:08.116292 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:08.126327 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:08.126307 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:08.126329 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:08.126609 5610 server_base.cc:1034] running on GCE node
I20260928 23:47:08.127189 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:08.127861 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:08.129043 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790639228129025 us; error 32 us; skew 500 ppm
Sep 28 23:47:08 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:08.131930 5610 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:47:08.132901 5610 webserver.cc:460] Webserver started at http://127.21.112.66:45463/ using document root <none> and password file <none>
I20260928 23:47:08.133486 5610 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:08.133607 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:08.137620 5610 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:47:08.139814 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:08.141080 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:08.141209 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "09865c7b6d1d413386b075b5ed927616"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "a82bf697d728f54339121c2d31e36040"
server_key_iv: "1a36944ea5ae0a6033b7dbc8310df546"
server_key_version: "encryptionkey@0"
I20260928 23:47:08.141826 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:08.154321 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:08.157508 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:08.157743 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:08.158295 5610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:08.159123 5610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:08.159169 5610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:08.159214 5610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:08.159235 5610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:08.168586 5610 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:42429
I20260928 23:47:08.168672 5743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:42429 every 8 connection(s)
I20260928 23:47:08.169821 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:08.175755 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5610
I20260928 23:47:08.176251 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:45473
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:36833
--webserver_port=44717
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:47:08 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:08.184435 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.171833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:53987 (local address 127.21.112.126:36833)
0928 23:47:08.171993 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.171997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.173302 (+ 1305us) server_negotiation.cc:407] Connection header received
0928 23:47:08.174154 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.174157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.174206 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.174286 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:08.175497 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.176300 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.178095 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.178333 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.178445 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.180877 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:08.180881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:08.180882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:08.180905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:08.182412 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.182912 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.182916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.182918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.182996 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.183343 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.183347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.183349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.183526 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:08.183631 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.184110 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.184249 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:47:08.185228 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.172234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:53987)
0928 23:47:08.173092 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.173144 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.173954 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.174457 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.174463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.174826 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:08.175350 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.175363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.176468 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.176474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.177972 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.177982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.178484 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.178488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.178661 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.179277 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:08.179293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:08.180707 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:08.182528 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:08.182534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:08.182548 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:08.182810 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:08.183092 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:08.183095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:08.183097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:08.183249 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:08.183626 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:08.183632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:08.183874 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.184058 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.184382 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":523,"threads_started":1}
I20260928 23:47:08.186623 5744 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36833
I20260928 23:47:08.186877 5744 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:08.187376 5744 heartbeater.cc:511] Master 127.21.112.126:36833 requested a full tablet report, sending...
I20260928 23:47:08.188416 5415 ts_manager.cc:194] Registered new tserver with Master: 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
I20260928 23:47:08.189165 5415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:53987
W20260928 23:47:08.291106 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:08.291414 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:08.291484 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:08.295902 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:08.295971 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:08.296032 5747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:08.301713 5747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:45473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=44717
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:08.302838 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:08.303828 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:47:08.314173 5747 server_base.cc:1034] running on GCE node
W20260928 23:47:08.314136 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:08.314129 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:08.314131 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:08.314858 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:08.315580 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:08.316769 5747 hybrid_clock.cc:648] HybridClock initialized: now 1790639228316749 us; error 38 us; skew 500 ppm
Sep 28 23:47:08 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:08.320384 5747 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:47:08.321523 5747 webserver.cc:460] Webserver started at http://127.21.112.67:44717/ using document root <none> and password file <none>
I20260928 23:47:08.322276 5747 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:08.322386 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:08.326648 5747 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 23:47:08.328835 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:08.330430 5747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:08.330575 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "c6d84b054ad7424583f17e2831a7b3c0"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "c154e662c0962b90ab31bca0858b480e"
server_key_iv: "13f35f9eed15643ee4679e1d509595d4"
server_key_version: "encryptionkey@0"
I20260928 23:47:08.331251 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:08.346364 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:08.349371 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:08.349617 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:08.350201 5747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:08.351007 5747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:08.351074 5747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:08.351152 5747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:08.351202 5747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:08.359915 5747 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:45473
I20260928 23:47:08.359935 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:45473 every 8 connection(s)
I20260928 23:47:08.360901 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:08.363170 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5747
Sep 28 23:47:08 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639228, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:08.372766 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.363840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.21.112.126:36833)
0928 23:47:08.364106 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.364111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.364308 (+ 197us) server_negotiation.cc:407] Connection header received
0928 23:47:08.364474 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.364477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.364526 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.364617 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:08.365350 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.365980 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.366834 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.367075 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.367189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.368602 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:08.368605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:08.368608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:08.368652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:08.370366 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.370902 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.370908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.370910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.371008 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.371655 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.371660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.371663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.371924 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:08.372166 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.372355 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.372554 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260928 23:47:08.375357 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.362741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43665 (local address 127.21.112.126:36833)
0928 23:47:08.362880 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.362884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.363806 (+ 922us) server_negotiation.cc:407] Connection header received
0928 23:47:08.365266 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.365270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.365317 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.365393 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:08.366634 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.367134 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.368263 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.368484 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.368585 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.371352 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:08.371356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:08.371359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:08.371390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:08.373055 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.373728 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.373731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.373732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.373838 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.374209 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.374213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.374214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.374389 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:08.374491 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.375075 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.375202 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":7424}
I20260928 23:47:08.376216 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.363152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:43665)
0928 23:47:08.363654 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.363707 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.365052 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.365614 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.365621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.365967 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:08.366441 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.366492 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.367259 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.367263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.368121 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.368129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.368640 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.368645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.368812 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.369375 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:08.369391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:08.371181 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:08.373181 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:08.373187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:08.373224 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:08.373615 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:08.373939 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:08.373942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:08.373943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:08.374094 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:08.374490 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:08.374498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:08.374826 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.375020 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.375357 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":148,"threads_started":1}
I20260928 23:47:08.377339 5876 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36833
I20260928 23:47:08.377629 5876 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:08.378132 5876 heartbeater.cc:511] Master 127.21.112.126:36833 requested a full tablet report, sending...
I20260928 23:47:08.379206 5415 ts_manager.cc:194] Registered new tserver with Master: c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473)
I20260928 23:47:08.379832 5415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:43665
I20260928 23:47:08.386787 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:08.391260 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.387554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.21.112.126:36833)
0928 23:47:08.387709 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.387721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.387848 (+ 127us) server_negotiation.cc:407] Connection header received
0928 23:47:08.387992 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.387995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.388110 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.388206 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:08.388469 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.389043 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.390012 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.390274 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.390366 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.390686 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:08.390828 (+ 142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 23:47:08.391151 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54320: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 23:47:08.391323 5608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54320: token expired
W20260928 23:47:08.391705 5348 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.112.126:36833)
I20260928 23:47:08.399376 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.392078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.21.112.126:36833)
0928 23:47:08.392186 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.392189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.392201 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:47:08.392282 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.392284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.392317 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.392380 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:08.393020 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.393694 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.394527 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.394776 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.394883 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.396253 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:08.396256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:08.396259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:08.396285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:08.397753 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.398180 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.398183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.398184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.398262 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:08.398533 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:08.398536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:08.398538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:08.398717 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:08.398849 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.399086 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.399212 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:47:08.401895 5415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:08.405164 5415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:08.413659 5540 tablet_service.cc:1467] Processing CreateTablet for tablet ba5fcc157b764b5cb4df336d6676919e (DEFAULT_TABLE table=test-table [id=175365f59a6f4c02a76c778f627c5e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:08.414371 5540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5fcc157b764b5cb4df336d6676919e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:08.416915 5498 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Bootstrap starting.
I20260928 23:47:08.417984 5498 tablet_bootstrap.cc:654] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:08.418509 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.412709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42429 (local address 127.0.0.1:46476)
0928 23:47:08.413328 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.413339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.413450 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.414098 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.414101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.414118 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:08.414205 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.414211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.415581 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.415584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.416771 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.416783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.417936 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.417943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.418055 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.418116 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.418216 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.418310 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":26,"thread_start_us":101,"threads_started":1}
I20260928 23:47:08.418529 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.412921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.21.112.66:42429)
0928 23:47:08.413221 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.413234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.413427 (+ 193us) server_negotiation.cc:407] Connection header received
0928 23:47:08.413595 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.413616 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.413843 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.413996 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:08.414320 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.415456 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.416921 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.417813 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.417931 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.418092 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.418249 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.418349 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":122,"threads_started":1}
I20260928 23:47:08.419020 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.412345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45473 (local address 127.0.0.1:35744)
0928 23:47:08.412479 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.412498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.412597 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.413648 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.413651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.413668 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:08.413785 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.413791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.415382 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.415386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.416775 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.416783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.418599 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.418604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.418736 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.418765 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.418820 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.418905 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":48}
I20260928 23:47:08.419796 5498 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: No bootstrap required, opened a new log
I20260928 23:47:08.419888 5498 ts_tablet_manager.cc:1397] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:08.419929 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.412624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.21.112.67:45473)
0928 23:47:08.413048 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.413060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.413086 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:47:08.413178 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.413184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.413419 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.413671 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:08.413886 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.415264 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.416921 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.418453 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.419466 (+ 1013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.419635 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.419685 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.419765 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":121,"threads_started":1}
I20260928 23:47:08.420339 5498 raft_consensus.cc:348] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.420481 5498 raft_consensus.cc:374] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:08.420531 5498 raft_consensus.cc:729] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03106721b4645abbc58c5d949a05dc5, State: Initialized, Role: FOLLOWER
I20260928 23:47:08.420652 5498 consensus_queue.cc:260] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.420967 5498 ts_tablet_manager.cc:1428] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:47:08.422077 5678 tablet_service.cc:1467] Processing CreateTablet for tablet ba5fcc157b764b5cb4df336d6676919e (DEFAULT_TABLE table=test-table [id=175365f59a6f4c02a76c778f627c5e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:08.423056 5678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5fcc157b764b5cb4df336d6676919e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:08.422803 5810 tablet_service.cc:1467] Processing CreateTablet for tablet ba5fcc157b764b5cb4df336d6676919e (DEFAULT_TABLE table=test-table [id=175365f59a6f4c02a76c778f627c5e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:08.423931 5810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5fcc157b764b5cb4df336d6676919e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:08.431332 5889 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Bootstrap starting.
I20260928 23:47:08.431260 5890 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Bootstrap starting.
I20260928 23:47:08.433379 5889 tablet_bootstrap.cc:654] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:08.433753 5890 tablet_bootstrap.cc:654] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:08.434501 5890 log.cc:824] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:08.434499 5889 log.cc:824] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:08.436501 5889 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: No bootstrap required, opened a new log
I20260928 23:47:08.436538 5890 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: No bootstrap required, opened a new log
I20260928 23:47:08.436765 5889 ts_tablet_manager.cc:1397] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:47:08.436794 5890 ts_tablet_manager.cc:1397] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 23:47:08.439803 5889 raft_consensus.cc:348] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.439803 5890 raft_consensus.cc:348] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.440042 5889 raft_consensus.cc:374] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:08.440066 5890 raft_consensus.cc:374] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:08.440105 5889 raft_consensus.cc:729] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d84b054ad7424583f17e2831a7b3c0, State: Initialized, Role: FOLLOWER
I20260928 23:47:08.440131 5890 raft_consensus.cc:729] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09865c7b6d1d413386b075b5ed927616, State: Initialized, Role: FOLLOWER
I20260928 23:47:08.440553 5890 consensus_queue.cc:260] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.440554 5889 consensus_queue.cc:260] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.441278 5744 heartbeater.cc:503] Master 127.21.112.126:36833 was elected leader, sending a full tablet report...
I20260928 23:47:08.441473 5876 heartbeater.cc:503] Master 127.21.112.126:36833 was elected leader, sending a full tablet report...
I20260928 23:47:08.441596 5889 ts_tablet_manager.cc:1428] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:47:08.441614 5890 ts_tablet_manager.cc:1428] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 23:47:08.472313 5607 tablet.cc:2369] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:08.545022 5612 raft_consensus.cc:482] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:08.545337 5612 raft_consensus.cc:504] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.546672 5612 leader_election.cc:288] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473), 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
I20260928 23:47:08.550981 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.547012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:35783 (local address 127.21.112.67:45473)
0928 23:47:08.547255 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.547263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.547350 (+ 87us) server_negotiation.cc:407] Connection header received
0928 23:47:08.547557 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.547560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.547612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.547698 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:08.547976 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.548561 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.549933 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.550325 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.550452 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.550506 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.550779 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.550853 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 23:47:08.550980 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.547255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:50913 (local address 127.21.112.66:42429)
0928 23:47:08.547373 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.547377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.547555 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:47:08.547703 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.547706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.547759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.547840 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:08.548150 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.548702 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.549907 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.550319 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.550450 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.550504 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.550766 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.550835 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:47:08.551002 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.547077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42429 (local address 127.21.112.65:50913)
0928 23:47:08.547444 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.547456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.547579 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.547870 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.547872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.547887 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:08.548007 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.548014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.548823 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.548826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.549784 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.549793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.550437 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.550440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.550571 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.550648 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.550709 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.550804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":55,"thread_start_us":65,"threads_started":1}
I20260928 23:47:08.551013 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.546882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45473 (local address 127.21.112.65:35783)
0928 23:47:08.547252 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.547269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:08.547418 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:08.547718 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:08.547721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:08.547737 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:08.547867 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.547874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.548690 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.548693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.549789 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:08.549799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.550460 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.550464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:08.550623 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.550657 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 23:47:08.550749 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:47:08.550827 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":116,"thread_start_us":130,"threads_started":1}
I20260928 23:47:08.551678 5698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5fcc157b764b5cb4df336d6676919e" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09865c7b6d1d413386b075b5ed927616" is_pre_election: true
I20260928 23:47:08.552069 5698 raft_consensus.cc:2454] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 0.
I20260928 23:47:08.552032 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5fcc157b764b5cb4df336d6676919e" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d84b054ad7424583f17e2831a7b3c0" is_pre_election: true
I20260928 23:47:08.552340 5830 raft_consensus.cc:2454] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 0.
I20260928 23:47:08.552615 5495 leader_election.cc:302] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09865c7b6d1d413386b075b5ed927616, d03106721b4645abbc58c5d949a05dc5; no voters:
I20260928 23:47:08.552956 5612 raft_consensus.cc:2781] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:08.553036 5612 raft_consensus.cc:482] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:08.553081 5612 raft_consensus.cc:3037] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:08.554175 5612 raft_consensus.cc:504] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.554603 5612 leader_election.cc:288] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 election: Requested vote from peers c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473), 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
I20260928 23:47:08.554963 5830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5fcc157b764b5cb4df336d6676919e" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d84b054ad7424583f17e2831a7b3c0"
I20260928 23:47:08.555064 5698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba5fcc157b764b5cb4df336d6676919e" candidate_uuid: "d03106721b4645abbc58c5d949a05dc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09865c7b6d1d413386b075b5ed927616"
I20260928 23:47:08.555140 5830 raft_consensus.cc:3037] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:08.555191 5698 raft_consensus.cc:3037] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:08.556171 5830 raft_consensus.cc:2454] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 1.
I20260928 23:47:08.556181 5698 raft_consensus.cc:2454] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03106721b4645abbc58c5d949a05dc5 in term 1.
I20260928 23:47:08.556493 5494 leader_election.cc:302] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6d84b054ad7424583f17e2831a7b3c0, d03106721b4645abbc58c5d949a05dc5; no voters:
I20260928 23:47:08.556661 5612 raft_consensus.cc:2781] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:08.556865 5612 raft_consensus.cc:686] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 LEADER]: Becoming Leader. State: Replica: d03106721b4645abbc58c5d949a05dc5, State: Running, Role: LEADER
I20260928 23:47:08.557134 5612 consensus_queue.cc:237] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:08.560209 5415 catalog_manager.cc:5495] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d03106721b4645abbc58c5d949a05dc5 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "d03106721b4645abbc58c5d949a05dc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } health_report { overall_health: HEALTHY } } }
I20260928 23:47:08.573630 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.569879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.21.112.65:42207)
0928 23:47:08.570247 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:08.570253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:08.570265 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:47:08.570337 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:08.570339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:08.570391 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:08.570521 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:08.570761 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.571272 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.572093 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:08.572319 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:08.572427 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:08.572705 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:08.573159 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:08.573289 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:08.573352 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 23:47:08.573411 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
I20260928 23:47:08.577816 5698 raft_consensus.cc:1261] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Refusing update from remote peer d03106721b4645abbc58c5d949a05dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:08.577812 5830 raft_consensus.cc:1261] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Refusing update from remote peer d03106721b4645abbc58c5d949a05dc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:08.578663 5897 consensus_queue.cc:1035] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:08.578871 5612 consensus_queue.cc:1035] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:08.586958 5902 mvcc.cc:205] Tried to move back new op lower bound from 7334458280247451648 to 7334458280172859392. Current Snapshot: MvccSnapshot[applied={T|T < 7334458280247451648}]
I20260928 23:47:08.588891 5901 mvcc.cc:205] Tried to move back new op lower bound from 7334458280247451648 to 7334458280172859392. Current Snapshot: MvccSnapshot[applied={T|T < 7334458280247451648}]
I20260928 23:47:08.594074 5415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:08.594259 5415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:08.596840 5415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Sending DeleteTablet for 3 replicas of tablet ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:08.597627 5678 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5fcc157b764b5cb4df336d6676919e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:08 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:46476
I20260928 23:47:08.597946 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5456
I20260928 23:47:08.597921 5810 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5fcc157b764b5cb4df336d6676919e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:08 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:35744
I20260928 23:47:08.598089 5540 tablet_service.cc:1514] Processing DeleteTablet for tablet ba5fcc157b764b5cb4df336d6676919e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:08 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51280
I20260928 23:47:08.598800 5909 tablet_replica.cc:330] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: stopping tablet replica
I20260928 23:47:08.599110 5909 raft_consensus.cc:2229] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:08.599419 5909 raft_consensus.cc:2258] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:08.600759 5909 ts_tablet_manager.cc:1905] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:08.603267 5909 ts_tablet_manager.cc:1918] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:08.603353 5909 log.cc:1196] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:08.603632 5910 tablet_replica.cc:330] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: stopping tablet replica
I20260928 23:47:08.603658 5909 ts_tablet_manager.cc:1939] T ba5fcc157b764b5cb4df336d6676919e P 09865c7b6d1d413386b075b5ed927616: Deleting consensus metadata
I20260928 23:47:08.604017 5910 raft_consensus.cc:2229] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:08.604408 5910 raft_consensus.cc:2258] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:08.604988 5405 catalog_manager.cc:4841] TS 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429): tablet ba5fcc157b764b5cb4df336d6676919e (table test-table [id=175365f59a6f4c02a76c778f627c5e30]) successfully deleted
W20260928 23:47:08.606067 5402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:42207 (error 108)
I20260928 23:47:08.606242 5910 ts_tablet_manager.cc:1905] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:08.607161 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:08.606839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:42207 (local address 127.0.0.1:51300)
0928 23:47:08.606988 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:08.607068 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:42207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:47:08.607285 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5610
W20260928 23:47:08.607334 5402 catalog_manager.cc:4584] TS d03106721b4645abbc58c5d949a05dc5 (127.21.112.65:42207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba5fcc157b764b5cb4df336d6676919e: Network error: Client connection negotiation failed: client connection to 127.21.112.65:42207: connect: Connection refused (error 111)
I20260928 23:47:08.609495 5910 ts_tablet_manager.cc:1918] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:08.609642 5910 log.cc:1196] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:08.610004 5910 ts_tablet_manager.cc:1939] T ba5fcc157b764b5cb4df336d6676919e P c6d84b054ad7424583f17e2831a7b3c0: Deleting consensus metadata
I20260928 23:47:08.611336 5404 catalog_manager.cc:4841] TS c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473): tablet ba5fcc157b764b5cb4df336d6676919e (table test-table [id=175365f59a6f4c02a76c778f627c5e30]) successfully deleted
I20260928 23:47:08.614144 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5747
I20260928 23:47:08.619953 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5385
I20260928 23:47:28.626927 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36833
--webserver_interface=127.21.112.126
--webserver_port=44267
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:28.742465 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:28.742762 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:28.742805 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:28.746871 5912 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:47:28.746933 5912 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:47:28.746950 5912 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:47:28.746964 5912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:28.746984 5912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:28.747040 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:28.747074 5912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:28.747094 5912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:28.752605 5912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=44267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:28.753804 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:28.755013 5912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:28.764765 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:28.764845 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:28.764778 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:28.764806 5912 server_base.cc:1034] running on GCE node
I20260928 23:47:28.765851 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:28.767073 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:28.768369 5912 hybrid_clock.cc:648] HybridClock initialized: now 1790639248768351 us; error 39 us; skew 500 ppm
Sep 28 23:47:28 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639248, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:28.771413 5912 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:47:28.772354 5912 webserver.cc:460] Webserver started at http://127.21.112.126:44267/ using document root <none> and password file <none>
I20260928 23:47:28.772940 5912 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:28.773003 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:28.777381 5912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:47:28.779747 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:28.781071 5912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:28.781184 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "5dd58ae18a114aa98974125f8c384192"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "4fc9519eb9fcc7f7d76dd69bd9e32572"
server_key_iv: "ddaf2015a0d6f3d5534d8cb74f9aed24"
server_key_version: "encryptionkey@0"
I20260928 23:47:28.781813 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:28.796940 5912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:28.800432 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:28.800623 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:28.808718 5912 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36833
I20260928 23:47:28.809008 5979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36833 every 8 connection(s)
I20260928 23:47:28.810052 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:28.814292 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5912
I20260928 23:47:28.814603 5980 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:47:28.814826 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:42207
--local_ip_for_outbound_sockets=127.21.112.65
--tserver_master_addrs=127.21.112.126:36833
--webserver_port=45187
--webserver_interface=127.21.112.65
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:47:28.816636 5980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap starting.
I20260928 23:47:28.842362 5980 log.cc:824] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:28.856864 5980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:47:28.857366 5980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Bootstrap complete.
I20260928 23:47:28.862967 5980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:28.863500 5980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd58ae18a114aa98974125f8c384192, State: Initialized, Role: FOLLOWER
I20260928 23:47:28.864066 5980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:28.864228 5980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:28.864334 5980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:28.864475 5980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:47:28.866876 5980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:28.867262 5980 leader_election.cc:302] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dd58ae18a114aa98974125f8c384192; no voters:
I20260928 23:47:28.867661 5980 leader_election.cc:288] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:47:28.867780 5984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:47:28.868021 5984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [term 3 LEADER]: Becoming Leader. State: Replica: 5dd58ae18a114aa98974125f8c384192, State: Running, Role: LEADER
I20260928 23:47:28.868433 5984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } }
I20260928 23:47:28.869037 5980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:28.869956 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5dd58ae18a114aa98974125f8c384192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } } }
I20260928 23:47:28.870117 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:28.870895 5986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd58ae18a114aa98974125f8c384192. Latest consensus state: current_term: 3 leader_uuid: "5dd58ae18a114aa98974125f8c384192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dd58ae18a114aa98974125f8c384192" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36833 } } }
I20260928 23:47:28.871023 5991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:28.871052 5986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:28.876739 5991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:28.877559 5991 catalog_manager.cc:1259] Loaded cluster ID: c181876993564f7c83d2ab89a2d16c35
I20260928 23:47:28.877624 5991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:28.880127 5991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:28.880846 5991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Loaded TSK: 0
I20260928 23:47:28.880916 5991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5dd58ae18a114aa98974125f8c384192: Loaded TSK: 1
F20260928 23:47:28.881666 5991 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6c543389ad google::LogMessage::Fail() at ??:0
@ 0x7f6c5433b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6c543384de google::LogMessage::Flush() at ??:0
@ 0x7f6c54339a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6c56ffc18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6c56ff4f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6c5ec9d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6c5ec7799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6c5ec78178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f6c5ecb9df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6c5ecd882b std::function<>::operator()() at ??:0
@ 0x7f6c5ec77b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6c5ec78f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6c5ec75579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6c5ecb9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6c5658a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6c5658ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6c5658c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6c5657b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6c57bbf6db start_thread at ??:0
@ 0x7f6c52e3661f clone at ??:0
W20260928 23:47:28.984676 5982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:28.984951 5982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:28.984995 5982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:28.989239 5982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:28.989310 5982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:28.989398 5982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:28.995234 5982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:42207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=45187
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:28.996373 5982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:28.997437 5982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:29.007486 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:29.007623 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:29.007627 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:29.007902 5982 server_base.cc:1034] running on GCE node
I20260928 23:47:29.008517 5982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:29.009263 5982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:29.010453 5982 hybrid_clock.cc:648] HybridClock initialized: now 1790639249010419 us; error 57 us; skew 500 ppm
Sep 28 23:47:29 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639249, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:29.013441 5982 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:47:29.014508 5982 webserver.cc:460] Webserver started at http://127.21.112.65:45187/ using document root <none> and password file <none>
I20260928 23:47:29.015120 5982 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:29.015214 5982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:29.019420 5982 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:47:29.021811 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.023365 5982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:29.023533 5982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "d03106721b4645abbc58c5d949a05dc5"
format_stamp: "Formatted at 2026-09-28 23:46:46 on dist-test-slave-c3s8"
server_key: "c8f3e91417810a0b5f190818eb066c6a"
server_key_iv: "5b1e92444aa082730d016b3cabff04f4"
server_key_version: "encryptionkey@0"
I20260928 23:47:29.024129 5982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:29.056349 5982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:29.059893 5982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:29.060110 5982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:29.060743 5982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:29.061964 6044 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:47:29.066247 5982 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:47:29.066303 5982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 23:47:29.066371 5982 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:47:29.068053 5982 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:47:29.068110 5982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 23:47:29.068361 6044 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Bootstrap starting.
I20260928 23:47:29.079094 5982 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:42207
I20260928 23:47:29.079421 6151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:42207 every 8 connection(s)
I20260928 23:47:29.080353 5982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:29.083618 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 5982
I20260928 23:47:29.084144 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:42429
--local_ip_for_outbound_sockets=127.21.112.66
--tserver_master_addrs=127.21.112.126:36833
--webserver_port=45463
--webserver_interface=127.21.112.66
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:47:29.089777 6044 log.cc:824] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:29.100028 6044 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:47:29.100468 6044 tablet_bootstrap.cc:492] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Bootstrap complete.
I20260928 23:47:29.101939 6044 ts_tablet_manager.cc:1397] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.015s
I20260928 23:47:29.105235 6044 raft_consensus.cc:348] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:29.105866 6044 raft_consensus.cc:729] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03106721b4645abbc58c5d949a05dc5, State: Initialized, Role: FOLLOWER
I20260928 23:47:29.106452 6044 consensus_queue.cc:260] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:29.107560 6044 ts_tablet_manager.cc:1428] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260928 23:47:29.115252 6154 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:47:29.082822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:42521)
0928 23:47:29.086081 (+ 3259us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.086152 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:29.087276 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:29.114449 (+ 27173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":179,"threads_started":1}
W20260928 23:47:29.115867 6152 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:36833 (0 consecutive failures): Network error: Failed to ping master at 127.21.112.126:36833: Client connection negotiation failed: client connection to 127.21.112.126:36833: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:47:29.116645 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.116363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:36885)
0928 23:47:29.116506 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.116588 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 23:47:29.117370 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.117160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:55927)
0928 23:47:29.117269 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.117323 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:47:29.117569 6152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:47:29.206075 6155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:29.206377 6155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:29.206452 6155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:29.210445 6155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:29.210536 6155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:29.210623 6155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:29.216102 6155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:42429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=45463
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:29.217263 6155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:29.218405 6155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:29.228219 6162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:29.228449 6155 server_base.cc:1034] running on GCE node
W20260928 23:47:29.228201 6163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:29.228205 6165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:29.229120 6155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:29.229837 6155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:29.231024 6155 hybrid_clock.cc:648] HybridClock initialized: now 1790639249231006 us; error 33 us; skew 500 ppm
Sep 28 23:47:29 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639249, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:29.233994 6155 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:47:29.234899 6155 webserver.cc:460] Webserver started at http://127.21.112.66:45463/ using document root <none> and password file <none>
I20260928 23:47:29.235489 6155 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:29.235584 6155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:29.239565 6155 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:47:29.241854 6172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.243287 6155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:47:29.243441 6155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "09865c7b6d1d413386b075b5ed927616"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "a82bf697d728f54339121c2d31e36040"
server_key_iv: "1a36944ea5ae0a6033b7dbc8310df546"
server_key_version: "encryptionkey@0"
I20260928 23:47:29.244071 6155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:29.293090 6155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:29.296877 6155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:29.297096 6155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:29.297755 6155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:29.298609 6155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:29.298655 6155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.298724 6155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:29.298758 6155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.307940 6155 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:42429
I20260928 23:47:29.307965 6285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:42429 every 8 connection(s)
I20260928 23:47:29.308951 6155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:29.312948 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.311118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:48299)
0928 23:47:29.311635 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.311752 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
W20260928 23:47:29.313418 6286 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:36833 (0 consecutive failures): Network error: Failed to ping master at 127.21.112.126:36833: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
I20260928 23:47:29.314108 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.313856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:52813)
0928 23:47:29.313991 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.314040 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 23:47:29.314172 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 6155
I20260928 23:47:29.314714 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:45473
--local_ip_for_outbound_sockets=127.21.112.67
--tserver_master_addrs=127.21.112.126:36833
--webserver_port=44717
--webserver_interface=127.21.112.67
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:47:29.314798 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.314589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:46645)
0928 23:47:29.314686 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.314726 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:47:29.315001 6286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:47:29.427347 6289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:29.427616 6289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:29.427693 6289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:29.431824 6289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:29.431881 6289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:29.431967 6289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:29.437258 6289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:45473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=44717
--tserver_master_addrs=127.21.112.126:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:29.438360 6289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:29.439417 6289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:29.449106 6297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:29.449112 6294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:29.449117 6295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:29.449404 6289 server_base.cc:1034] running on GCE node
I20260928 23:47:29.449908 6289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:29.450629 6289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:29.451828 6289 hybrid_clock.cc:648] HybridClock initialized: now 1790639249451806 us; error 40 us; skew 500 ppm
Sep 28 23:47:29 dist-test-slave-c3s8 krb5kdc[4838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639249, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:29.454543 6289 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:47:29.455423 6289 webserver.cc:460] Webserver started at http://127.21.112.67:44717/ using document root <none> and password file <none>
I20260928 23:47:29.455950 6289 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:29.456007 6289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:29.459815 6289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 23:47:29.461901 6304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.463137 6289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:29.463263 6289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "c6d84b054ad7424583f17e2831a7b3c0"
format_stamp: "Formatted at 2026-09-28 23:46:47 on dist-test-slave-c3s8"
server_key: "c154e662c0962b90ab31bca0858b480e"
server_key_iv: "13f35f9eed15643ee4679e1d509595d4"
server_key_version: "encryptionkey@0"
I20260928 23:47:29.463837 6289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:29.479527 6289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:29.482755 6289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:29.482959 6289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:29.483556 6289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:29.484365 6289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:29.484411 6289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.484477 6289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:29.484510 6289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:29.493608 6289 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:45473
I20260928 23:47:29.493625 6417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:45473 every 8 connection(s)
I20260928 23:47:29.494745 6289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:29.498554 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.496777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:36659)
0928 23:47:29.497251 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.497358 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
W20260928 23:47:29.499065 6418 heartbeater.cc:644] Failed to heartbeat to 127.21.112.126:36833 (0 consecutive failures): Network error: Failed to ping master at 127.21.112.126:36833: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
I20260928 23:47:29.499588 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.499384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:44501)
0928 23:47:29.499490 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.499535 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 23:47:29.500195 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:29.499988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:45621)
0928 23:47:29.500100 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:29.500149 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:47:29.500356 6418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:47:29.501432 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 6289
I20260928 23:47:30.118764 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.118210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:44229)
0928 23:47:30.118591 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:30.118655 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":138,"threads_started":1}
I20260928 23:47:30.316160 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.315708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:38931)
0928 23:47:30.315991 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:30.316050 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260928 23:47:30.501348 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.500915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:50675)
0928 23:47:30.501172 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:30.501232 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260928 23:47:30.893769 6425 raft_consensus.cc:482] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:30.894009 6425 raft_consensus.cc:504] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:30.895394 6425 leader_election.cc:288] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473), 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
Sep 28 23:47:30 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639249, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.66@KRBTEST.COM
Sep 28 23:47:30 dist-test-slave-c3s8 krb5kdc[4838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639249, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.67@KRBTEST.COM
I20260928 23:47:30.909981 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.895857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:37991 (local address 127.21.112.67:45473)
0928 23:47:30.896307 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:30.896326 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:30.896358 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:47:30.897016 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:30.897022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:30.897365 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:30.897728 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:30.899018 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.900251 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.901532 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.902024 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.902195 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:30.905373 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:30.905377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:30.905388 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:30.905416 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:30.907368 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:30.907959 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:30.907965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:30.907977 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:30.908091 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:30.908434 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:30.908438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:30.908440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:30.908840 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:30.908974 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:30.909511 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 23:47:30.909794 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":54,"threads_started":1}
I20260928 23:47:30.909969 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.895698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45473 (local address 127.21.112.65:37991)
0928 23:47:30.896092 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:30.896107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:30.896204 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:30.897788 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:30.897825 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:30.898368 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:30.898836 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:30.898851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.900392 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.900397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:30.901400 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:30.901410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.902189 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.902193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:30.902325 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:30.903217 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:30.903229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:30.905158 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:30.907513 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:30.907518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:30.907532 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:30.907841 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:30.908196 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:30.908199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:30.908201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:30.908337 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:30.908964 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:30.908970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:30.909208 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 23:47:30.909401 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 23:47:30.909775 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":146,"spinlock_wait_cycles":27136,"thread_start_us":88,"threads_started":1}
I20260928 23:47:30.910076 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.895911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:52773 (local address 127.21.112.66:42429)
0928 23:47:30.896286 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:30.896309 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:30.896346 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:47:30.897016 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:30.897022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:30.897399 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:30.897725 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:30.899019 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.900271 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.901525 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.902023 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.902196 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:30.905262 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:30.905268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:30.905280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:30.905311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:30.907344 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:30.908010 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:30.908023 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:30.908028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:30.908153 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:30.908473 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:30.908478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:30.908480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:30.908870 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:30.909013 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:30.909526 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 23:47:30.909862 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":59,"threads_started":1}
I20260928 23:47:30.910368 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:30.895641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:42429 (local address 127.21.112.65:52773)
0928 23:47:30.896020 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:30.896035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:30.896168 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:30.897787 (+ 1619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:30.897793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:30.898368 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:30.898836 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:30.898850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.900395 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.900400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:30.901398 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:30.901408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:30.902147 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:30.902151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:30.902322 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:30.903106 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:30.903131 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:30.904809 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:30.907528 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:30.907534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:30.907536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:30.907869 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:30.908271 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:30.908279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:30.908280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:30.908372 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:30.908996 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:30.909000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:30.909209 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 23:47:30.909402 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 23:47:30.909871 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":343,"spinlock_wait_cycles":3584,"thread_start_us":64,"threads_started":1}
W20260928 23:47:30.912961 6040 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
W20260928 23:47:30.913138 6041 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:30.913256 6041 leader_election.cc:302] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d03106721b4645abbc58c5d949a05dc5; no voters: 09865c7b6d1d413386b075b5ed927616, c6d84b054ad7424583f17e2831a7b3c0
I20260928 23:47:30.913583 6425 raft_consensus.cc:2726] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:47:31.120129 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:31.119745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:59587)
0928 23:47:31.119943 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:31.120015 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 23:47:31.317482 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:31.316978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:55637)
0928 23:47:31.317308 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:31.317373 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260928 23:47:31.502574 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:31.502128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:38371)
0928 23:47:31.502424 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:31.502484 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260928 23:47:32.121646 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:32.121020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:44513)
0928 23:47:32.121366 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:32.121488 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":126,"threads_started":1}
I20260928 23:47:32.318812 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:32.318309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:42551)
0928 23:47:32.318655 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:32.318720 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260928 23:47:32.503839 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:32.503398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:58153)
0928 23:47:32.503667 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:32.503728 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260928 23:47:32.630332 6435 raft_consensus.cc:482] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:32.630468 6435 raft_consensus.cc:504] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:32.630988 6435 leader_election.cc:288] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473), 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
W20260928 23:47:32.631827 6040 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
W20260928 23:47:32.631934 6041 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:32.632061 6041 leader_election.cc:302] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d03106721b4645abbc58c5d949a05dc5; no voters: 09865c7b6d1d413386b075b5ed927616, c6d84b054ad7424583f17e2831a7b3c0
I20260928 23:47:32.632256 6435 raft_consensus.cc:2726] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:47:33.123059 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:33.122532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:50677)
0928 23:47:33.122864 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:33.122935 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
I20260928 23:47:33.320142 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:33.319678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:52439)
0928 23:47:33.319969 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:33.320029 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260928 23:47:33.505010 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:33.504619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:36765)
0928 23:47:33.504853 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:33.504909 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260928 23:47:34.124560 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:34.124099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:52675)
0928 23:47:34.124406 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:34.124462 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":104,"threads_started":1}
I20260928 23:47:34.321676 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:34.321089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:33149)
0928 23:47:34.321413 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:34.321497 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260928 23:47:34.470952 6441 raft_consensus.cc:482] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:34.471088 6441 raft_consensus.cc:504] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:34.471694 6441 leader_election.cc:288] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473), 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
W20260928 23:47:34.472488 6040 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
W20260928 23:47:34.472604 6041 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:34.472728 6041 leader_election.cc:302] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d03106721b4645abbc58c5d949a05dc5; no voters: 09865c7b6d1d413386b075b5ed927616, c6d84b054ad7424583f17e2831a7b3c0
I20260928 23:47:34.472898 6441 raft_consensus.cc:2726] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:47:34.506351 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:34.505929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:43331)
0928 23:47:34.506190 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:34.506256 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260928 23:47:35.126055 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:35.125500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:53237)
0928 23:47:35.125888 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:35.125955 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
I20260928 23:47:35.322963 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:35.322549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:59929)
0928 23:47:35.322804 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:35.322866 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260928 23:47:35.507480 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:35.507086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:49637)
0928 23:47:35.507324 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:35.507387 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260928 23:47:36.127501 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:36.126909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:36137)
0928 23:47:36.127327 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:36.127395 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":74,"threads_started":1}
I20260928 23:47:36.324249 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:36.323807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:34119)
0928 23:47:36.324099 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:36.324155 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260928 23:47:36.508725 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:36.508245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:37939)
0928 23:47:36.508571 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:36.508630 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260928 23:47:37.128703 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:37.128279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:50771)
0928 23:47:37.128545 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:37.128599 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260928 23:47:37.325625 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:37.325108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:33085)
0928 23:47:37.325413 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:37.325472 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":80,"threads_started":1}
I20260928 23:47:37.510178 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:37.509745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:59945)
0928 23:47:37.510019 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:37.510081 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260928 23:47:37.850404 6452 raft_consensus.cc:482] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:37.850540 6452 raft_consensus.cc:504] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d84b054ad7424583f17e2831a7b3c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45473 } } peers { permanent_uuid: "09865c7b6d1d413386b075b5ed927616" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 42429 } } peers { permanent_uuid: "d03106721b4645abbc58c5d949a05dc5" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 42207 } }
I20260928 23:47:37.850941 6452 leader_election.cc:288] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473), 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429)
W20260928 23:47:37.851980 6041 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 09865c7b6d1d413386b075b5ed927616 (127.21.112.66:42429): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
W20260928 23:47:37.852154 6040 leader_election.cc:341] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6d84b054ad7424583f17e2831a7b3c0 (127.21.112.67:45473): Not found: Tablet not found: ba5fcc157b764b5cb4df336d6676919e
I20260928 23:47:37.852238 6040 leader_election.cc:302] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d03106721b4645abbc58c5d949a05dc5; no voters: 09865c7b6d1d413386b075b5ed927616, c6d84b054ad7424583f17e2831a7b3c0
I20260928 23:47:37.852442 6452 raft_consensus.cc:2726] T ba5fcc157b764b5cb4df336d6676919e P d03106721b4645abbc58c5d949a05dc5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:47:38.130172 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:38.129520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:34599)
0928 23:47:38.130009 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:38.130072 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":79,"threads_started":1}
I20260928 23:47:38.326864 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:38.326417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:52565)
0928 23:47:38.326690 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:38.326756 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260928 23:47:38.511387 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:38.510906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:33381)
0928 23:47:38.511165 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:38.511285 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260928 23:47:39.131511 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:39.131110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:51533)
0928 23:47:39.131356 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:39.131414 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260928 23:47:39.328281 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:39.327823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:37823)
0928 23:47:39.328113 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:39.328181 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260928 23:47:39.512763 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:39.512258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:47029)
0928 23:47:39.512586 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:39.512655 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260928 23:47:40.132805 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:40.132346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:33299)
0928 23:47:40.132633 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:40.132702 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260928 23:47:40.329641 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:40.329137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:50177)
0928 23:47:40.329417 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:40.329482 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260928 23:47:40.514289 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:40.513835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:36607)
0928 23:47:40.514114 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:40.514187 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260928 23:47:41.134297 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:41.133743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:43573)
0928 23:47:41.134053 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:41.134120 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20260928 23:47:41.330905 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:41.330458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:45493)
0928 23:47:41.330728 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:41.330798 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260928 23:47:41.515496 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:41.515061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:59917)
0928 23:47:41.515345 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:41.515398 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260928 23:47:42.135676 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:42.135245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:49517)
0928 23:47:42.135520 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:42.135578 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260928 23:47:42.332201 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:42.331747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:45465)
0928 23:47:42.332019 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:42.332106 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260928 23:47:42.516813 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:42.516342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:38873)
0928 23:47:42.516629 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:42.516695 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":126,"threads_started":1}
I20260928 23:47:43.137022 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:43.136576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:40389)
0928 23:47:43.136863 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:43.136925 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260928 23:47:43.333489 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:43.333069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:56221)
0928 23:47:43.333336 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:43.333392 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260928 23:47:43.518383 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:43.517896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:41837)
0928 23:47:43.518219 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:43.518283 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260928 23:47:44.138638 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:44.138199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.65:45013)
0928 23:47:44.138466 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:44.138539 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":140,"threads_started":1}
I20260928 23:47:44.335001 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:44.334543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.66:45815)
0928 23:47:44.334828 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:44.334891 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260928 23:47:44.512607 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 5982
I20260928 23:47:44.518541 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 6155
I20260928 23:47:44.520248 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:44.519664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36833 (local address 127.21.112.67:57321)
0928 23:47:44.520042 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:44.520110 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.126:36833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":78,"threads_started":1}
I20260928 23:47:44.524334 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 6289
2026-09-28T23:47:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59337 ms)
[----------] 30 tests from SecurityITest (249564 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:44 dist-test-slave-c3s8 krb5kdc[6476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:44 dist-test-slave-c3s8 krb5kdc[6476](info): set up 2 sockets
Sep 28 23:47:44 dist-test-slave-c3s8 krb5kdc[6476](info): commencing operation
krb5kdc: starting...
W20260928 23:47:45.566000 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:47:45 dist-test-slave-c3s8 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:47:45.693133 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46281
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:45.806906 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:45.807221 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:45.807327 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:45.811424 6489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:45.811509 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:45.811547 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:45.811604 6489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:45.811659 6489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:45.817191 6489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:45.818462 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:45.819597 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:45.829164 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:45.829303 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:45.829372 6489 server_base.cc:1034] running on GCE node
W20260928 23:47:45.829303 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:45.830113 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:45.831241 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:45.832418 6489 hybrid_clock.cc:648] HybridClock initialized: now 1790639265832401 us; error 28 us; skew 500 ppm
Sep 28 23:47:45 dist-test-slave-c3s8 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639265, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:45.835367 6489 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:47:45.836303 6489 webserver.cc:460] Webserver started at http://127.21.112.126:44231/ using document root <none> and password file <none>
I20260928 23:47:45.836886 6489 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:45.836978 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:45.837214 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:45.839406 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "f6513c39beaf471d909cacf417b27daf"
format_stamp: "Formatted at 2026-09-28 23:47:45 on dist-test-slave-c3s8"
server_key: "020e6727bce7f380bddfba195089d1aa"
server_key_iv: "cc16972d4f542fda2936d41b09f8520d"
server_key_version: "encryptionkey@0"
I20260928 23:47:45.839974 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "f6513c39beaf471d909cacf417b27daf"
format_stamp: "Formatted at 2026-09-28 23:47:45 on dist-test-slave-c3s8"
server_key: "020e6727bce7f380bddfba195089d1aa"
server_key_iv: "cc16972d4f542fda2936d41b09f8520d"
server_key_version: "encryptionkey@0"
I20260928 23:47:45.843845 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:47:45.846516 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:45.847877 6489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:45.848024 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "f6513c39beaf471d909cacf417b27daf"
format_stamp: "Formatted at 2026-09-28 23:47:45 on dist-test-slave-c3s8"
server_key: "020e6727bce7f380bddfba195089d1aa"
server_key_iv: "cc16972d4f542fda2936d41b09f8520d"
server_key_version: "encryptionkey@0"
I20260928 23:47:45.848145 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:45.863834 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:45.866900 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:45.867107 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:45.874285 6489 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:46281
I20260928 23:47:45.874311 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:46281 every 8 connection(s)
I20260928 23:47:45.875207 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:45.877960 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:45.879866 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 6489
I20260928 23:47:45.880028 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:45.880402 21953 external_mini_cluster.cc:1428] Setting key 28244d0d96cdd9aa97f590337aa3fb80
I20260928 23:47:45.883858 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: Bootstrap starting.
I20260928 23:47:45.886662 6557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:45.887424 6557 log.cc:824] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:45.889909 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: No bootstrap required, opened a new log
Sep 28 23:47:45 dist-test-slave-c3s8 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:45.893282 6557 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6513c39beaf471d909cacf417b27daf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46281 } }
I20260928 23:47:45.893533 6557 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:45.893643 6557 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6513c39beaf471d909cacf417b27daf, State: Initialized, Role: FOLLOWER
I20260928 23:47:45.894091 6557 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6513c39beaf471d909cacf417b27daf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46281 } }
I20260928 23:47:45.894271 6557 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:45.894356 6557 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:45.894472 6557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:45.895424 6557 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6513c39beaf471d909cacf417b27daf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46281 } }
I20260928 23:47:45.895787 6557 leader_election.cc:302] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6513c39beaf471d909cacf417b27daf; no voters:
I20260928 23:47:45.896118 6557 leader_election.cc:288] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:45.896248 6562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:45.896450 6562 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [term 1 LEADER]: Becoming Leader. State: Replica: f6513c39beaf471d909cacf417b27daf, State: Running, Role: LEADER
I20260928 23:47:45.896505 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:45.881847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.21.112.126:46281)
0928 23:47:45.882316 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:45.882333 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:45.882368 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:47:45.883057 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:45.883074 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:45.883445 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:45.883842 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:45.884863 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:45.886209 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:45.887177 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:45.887729 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:45.887914 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:45.890786 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:45.890792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:45.890805 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:45.890832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:45.893003 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:45.893554 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:45.893562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:45.893569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:45.893749 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:45.894083 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:45.894086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:45.894088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:45.894667 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:45.894890 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:45.895149 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 23:47:45.895458 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260928 23:47:45.897181 6562 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6513c39beaf471d909cacf417b27daf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46281 } }
I20260928 23:47:45.897439 6557 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:45.899289 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6513c39beaf471d909cacf417b27daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6513c39beaf471d909cacf417b27daf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46281 } } }
I20260928 23:47:45.899442 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:45.899801 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6513c39beaf471d909cacf417b27daf. Latest consensus state: current_term: 1 leader_uuid: "f6513c39beaf471d909cacf417b27daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6513c39beaf471d909cacf417b27daf" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46281 } } }
I20260928 23:47:45.899897 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf [sys.catalog]: This master's current role is: LEADER
W20260928 23:47:45.902823 6576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:47:45.902904 6576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:47:45.903047 6577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:45.903797 6577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:45.909726 6577 catalog_manager.cc:1347] Generated new cluster ID: 77694bf47bd247e29fc71e429d5766b6
I20260928 23:47:45.909834 6577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:45.926661 6577 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:45.927963 6577 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:45.934435 6577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: Generated new TSK 0
I20260928 23:47:45.935122 6577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:47:45.965703 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46281
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:46.085043 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:46.085332 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:46.085377 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:46.089522 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:46.089618 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:46.089740 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:46.095469 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:46.096632 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:46.097802 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:46.108049 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:46.108083 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:46.108141 6583 server_base.cc:1034] running on GCE node
W20260928 23:47:46.108367 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:46.108794 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:46.109468 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:46.110661 6583 hybrid_clock.cc:648] HybridClock initialized: now 1790639266110623 us; error 51 us; skew 500 ppm
Sep 28 23:47:46 dist-test-slave-c3s8 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639266, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:46.113693 6583 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:47:46.114614 6583 webserver.cc:460] Webserver started at http://127.21.112.65:33415/ using document root <none> and password file <none>
I20260928 23:47:46.115199 6583 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:46.115304 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:46.115543 6583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:46.117599 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "e38172be2eae484785c5e731726ef19f"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "780b7af1eba118e8b7275844b98d8830"
server_key_iv: "ab8eff8dffc2784cd0127fc2b40a2887"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.118165 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "e38172be2eae484785c5e731726ef19f"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "780b7af1eba118e8b7275844b98d8830"
server_key_iv: "ab8eff8dffc2784cd0127fc2b40a2887"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.122048 6583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:47:46.124718 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.126122 6583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:46.126260 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "e38172be2eae484785c5e731726ef19f"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "780b7af1eba118e8b7275844b98d8830"
server_key_iv: "ab8eff8dffc2784cd0127fc2b40a2887"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.126381 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:46.141047 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:46.144281 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:46.144490 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:46.145079 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:46.146005 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:46.146078 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.146152 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:46.146202 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.156399 6583 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:35557
I20260928 23:47:46.156424 6711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:35557 every 8 connection(s)
I20260928 23:47:46.157333 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:46.162770 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 6583
I20260928 23:47:46.162916 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:47:46.163220 21953 external_mini_cluster.cc:1428] Setting key 522150dbc18b32c29d0d726e93a7a21a
Sep 28 23:47:46 dist-test-slave-c3s8 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639266, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:46.172866 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.159211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:48037 (local address 127.21.112.126:46281)
0928 23:47:46.159576 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.159580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.160282 (+ 702us) server_negotiation.cc:407] Connection header received
0928 23:47:46.161728 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.161732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.161806 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.161920 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:46.163380 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.164038 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.165511 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.165955 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.166073 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.168671 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:46.168677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:46.168684 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:46.168718 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:46.170583 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.171138 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.171143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.171147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.171248 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.171604 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.171609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.171612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.171881 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:46.171988 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.172493 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.172654 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260928 23:47:46.173702 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.159651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46281 (local address 127.21.112.65:48037)
0928 23:47:46.160108 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.160150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.161464 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.162106 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.162113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.162562 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:46.163176 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.163193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.164175 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.164178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.165364 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.165374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.166116 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.166119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.166267 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.166949 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:46.166966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:46.168490 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:46.170716 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:46.170721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:46.170735 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:46.171028 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:46.171358 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:46.171360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:46.171362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:46.171504 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:46.171989 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:46.171994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:46.172241 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.172405 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.172674 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:47:46.175074 6712 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46281
I20260928 23:47:46.175403 6712 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:46.175953 6712 heartbeater.cc:511] Master 127.21.112.126:46281 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
I20260928 23:47:46.177703 6520 ts_manager.cc:194] Registered new tserver with Master: e38172be2eae484785c5e731726ef19f (127.21.112.65:35557)
I20260928 23:47:46.179268 6520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:48037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:47:46.193719 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46281
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:46.305279 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:46.305593 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:46.305709 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:46.309906 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:46.309988 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:46.310077 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:46.315335 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:46.316428 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:46.317452 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:46.327263 6723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:46.327263 6725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:46.327383 6722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:46.327445 6717 server_base.cc:1034] running on GCE node
I20260928 23:47:46.327975 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:46.328572 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:46.329734 6717 hybrid_clock.cc:648] HybridClock initialized: now 1790639266329711 us; error 29 us; skew 500 ppm
Sep 28 23:47:46 dist-test-slave-c3s8 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639266, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:46.332623 6717 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:47:46.333577 6717 webserver.cc:460] Webserver started at http://127.21.112.66:39193/ using document root <none> and password file <none>
I20260928 23:47:46.334192 6717 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:46.334285 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:46.334517 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:46.336541 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "1e7e507065a04040b1d8d79df3759702"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "6bd37275b2c187476c3e696d2ae0be1a"
server_key_iv: "bb952a126fe59430ba6c3e681e6a2e20"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.337146 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "1e7e507065a04040b1d8d79df3759702"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "6bd37275b2c187476c3e696d2ae0be1a"
server_key_iv: "bb952a126fe59430ba6c3e681e6a2e20"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.340906 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:47:46.343436 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.344725 6717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:47:46.344877 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "1e7e507065a04040b1d8d79df3759702"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "6bd37275b2c187476c3e696d2ae0be1a"
server_key_iv: "bb952a126fe59430ba6c3e681e6a2e20"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.344990 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:46.358263 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:46.361701 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:46.361909 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:46.362525 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:46.363384 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:46.363457 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.363530 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:46.363566 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.372586 6717 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:32779
I20260928 23:47:46.372609 6845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:32779 every 8 connection(s)
I20260928 23:47:46.373525 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:46.380396 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 6717
I20260928 23:47:46.380533 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:47:46.380823 21953 external_mini_cluster.cc:1428] Setting key 41f9585f98ebad6d4614434700ca9430
Sep 28 23:47:46 dist-test-slave-c3s8 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639266, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:46.388549 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.375489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:40331 (local address 127.21.112.126:46281)
0928 23:47:46.375629 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.375633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.376515 (+ 882us) server_negotiation.cc:407] Connection header received
0928 23:47:46.377942 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.377946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.378010 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.378118 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:46.379317 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.379856 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.381149 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.381440 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.381575 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.384522 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:46.384527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:46.384531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:46.384564 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:46.386336 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.386947 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.386951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.386953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.387041 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.387411 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.387416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.387418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.387612 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:46.387721 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.388253 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.388391 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:47:46.389609 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.375869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46281 (local address 127.21.112.66:40331)
0928 23:47:46.376348 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.376393 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.377720 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.378295 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.378302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.378647 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:46.379160 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.379173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.379989 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.379998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.381032 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.381041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.381642 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.381646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.381865 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.382582 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:46.382600 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:46.384340 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:46.386492 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:46.386497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:46.386512 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:46.386805 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:46.387148 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:46.387151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:46.387153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:46.387310 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:46.387731 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:46.387739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:46.388010 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.388212 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.388730 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:47:46.390841 6846 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46281
I20260928 23:47:46.391134 6846 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:46.391649 6846 heartbeater.cc:511] Master 127.21.112.126:46281 requested a full tablet report, sending...
I20260928 23:47:46.392808 6520 ts_manager.cc:194] Registered new tserver with Master: 1e7e507065a04040b1d8d79df3759702 (127.21.112.66:32779)
I20260928 23:47:46.393450 6520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:40331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:47:46.413622 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46281
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:46.532472 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:46.532717 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:46.532759 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:46.536857 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:46.536916 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:46.537015 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:46.542407 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:46.543509 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:46.544529 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:46.554559 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:46.554518 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:46.554569 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:46.554782 6851 server_base.cc:1034] running on GCE node
I20260928 23:47:46.555244 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:46.555876 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:46.557070 6851 hybrid_clock.cc:648] HybridClock initialized: now 1790639266557033 us; error 56 us; skew 500 ppm
Sep 28 23:47:46 dist-test-slave-c3s8 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639266, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:46.559926 6851 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:47:46.560757 6851 webserver.cc:460] Webserver started at http://127.21.112.67:34339/ using document root <none> and password file <none>
I20260928 23:47:46.561266 6851 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:46.561323 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:46.561480 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:46.563544 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "93c689cbda104f1f9eb01e88b237d819"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "d6fbbaf1e586dcf584e64ad644caa569"
server_key_iv: "a4136d34a591d727bcfa4cc6a71a6810"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.564041 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "93c689cbda104f1f9eb01e88b237d819"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "d6fbbaf1e586dcf584e64ad644caa569"
server_key_iv: "a4136d34a591d727bcfa4cc6a71a6810"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.567807 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:47:46.570194 6866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.571502 6851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:47:46.571607 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "93c689cbda104f1f9eb01e88b237d819"
format_stamp: "Formatted at 2026-09-28 23:47:46 on dist-test-slave-c3s8"
server_key: "d6fbbaf1e586dcf584e64ad644caa569"
server_key_iv: "a4136d34a591d727bcfa4cc6a71a6810"
server_key_version: "encryptionkey@0"
I20260928 23:47:46.571779 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:46.614246 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:46.617666 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:46.617894 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:46.618573 6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:46.619529 6851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:46.619611 6851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.619663 6851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:46.619685 6851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:46.629383 6851 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:34783
I20260928 23:47:46.629416 6979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:34783 every 8 connection(s)
I20260928 23:47:46.630551 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:46.630786 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 6851
I20260928 23:47:46.630936 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:47:46.631285 21953 external_mini_cluster.cc:1428] Setting key fcd190dbcfacf6dfaecc60fc6ee08f43
Sep 28 23:47:46 dist-test-slave-c3s8 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639266, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:46.644889 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.632586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:56239 (local address 127.21.112.126:46281)
0928 23:47:46.632728 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.632733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.633794 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 23:47:46.635184 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.635188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.635245 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.635325 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:46.636523 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.637058 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.638426 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.638653 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.638764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.641437 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:46.641440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:46.641442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:46.641466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:46.642753 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.643262 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.643266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.643268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.643354 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.643732 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.643735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.643737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.643921 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:46.644029 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.644594 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.644725 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:47:46.645865 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.633065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46281 (local address 127.21.112.67:56239)
0928 23:47:46.633656 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.633700 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.634976 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.635512 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.635520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.635886 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:46.636377 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.636390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.637186 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.637190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.638298 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.638309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.638805 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.638809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.638996 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.639654 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:46.639671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:46.641244 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:46.642873 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:46.642879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:46.642902 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:46.643153 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:46.643479 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:46.643484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:46.643486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:46.643624 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:46.644026 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:46.644033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:46.644327 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.644554 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.644878 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":157,"threads_started":1}
I20260928 23:47:46.646997 6980 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46281
I20260928 23:47:46.647214 6980 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:46.647670 6980 heartbeater.cc:511] Master 127.21.112.126:46281 requested a full tablet report, sending...
I20260928 23:47:46.648663 6520 ts_manager.cc:194] Registered new tserver with Master: 93c689cbda104f1f9eb01e88b237d819 (127.21.112.67:34783)
I20260928 23:47:46.649274 6520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:56239
I20260928 23:47:46.657325 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:46.667107 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.659444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.21.112.126:46281)
0928 23:47:46.659623 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.659628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.659781 (+ 153us) server_negotiation.cc:407] Connection header received
0928 23:47:46.659950 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.659953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.660002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.660083 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:46.660780 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.661357 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.662295 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.662507 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.662613 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.664021 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:46.664025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:46.664028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:46.664054 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:46.665367 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.665846 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.665850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.665852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.665931 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:46.666223 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:46.666227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:46.666228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:46.666445 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:46.666587 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.666828 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.666942 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 23:47:46.669965 6520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:46.672909 6520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:46.688257 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.681902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:34783 (local address 127.0.0.1:51968)
0928 23:47:46.683126 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.683183 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.683314 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.684121 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.684124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.684143 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:46.684257 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.684267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.685445 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.685452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.686807 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.686817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.687682 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.687685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.687798 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.687841 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.687963 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.688098 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260928 23:47:46.688249 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.682100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.21.112.67:34783)
0928 23:47:46.683602 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.683613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.683636 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:47:46.683725 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.683730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.683958 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.684111 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:46.684396 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.685314 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.686937 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.687543 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.687691 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.687871 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.687992 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.688090 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1392,"thread_start_us":204,"threads_started":1}
I20260928 23:47:46.689240 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.682966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:32779 (local address 127.0.0.1:36476)
0928 23:47:46.684198 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.684225 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.684330 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.684779 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.684782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.684801 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:46.684878 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.684883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.686318 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.686322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.687498 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.687508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.688798 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.688802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.688909 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.688931 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.689021 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.689101 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":32,"thread_start_us":108,"threads_started":1}
I20260928 23:47:46.689359 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.683401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.21.112.66:32779)
0928 23:47:46.683824 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.683834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.684330 (+ 496us) server_negotiation.cc:407] Connection header received
0928 23:47:46.684449 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.684454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.684625 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.684784 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:46.684992 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.686169 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.687636 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.688679 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.688769 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.688909 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.689080 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.689216 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":120,"threads_started":1}
I20260928 23:47:46.691352 6914 tablet_service.cc:1467] Processing CreateTablet for tablet e0d7e303b2014db4ba13bff0bea900d2 (DEFAULT_TABLE table=test-table [id=e6570a7a2f4a4c008e605629a9ee8d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:46.691632 6780 tablet_service.cc:1467] Processing CreateTablet for tablet e0d7e303b2014db4ba13bff0bea900d2 (DEFAULT_TABLE table=test-table [id=e6570a7a2f4a4c008e605629a9ee8d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:46.692406 6914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d7e303b2014db4ba13bff0bea900d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:46.692616 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d7e303b2014db4ba13bff0bea900d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:46.693967 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.681986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:35557 (local address 127.0.0.1:39798)
0928 23:47:46.683126 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.683184 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.683363 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.683979 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.683985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.684010 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:46.684165 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.684172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.685740 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.685744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.687001 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.687011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.688382 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.688386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.688473 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.693597 (+ 5124us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.693682 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.693788 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":54,"thread_start_us":117,"threads_started":1}
I20260928 23:47:46.694485 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.682599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.21.112.65:35557)
0928 23:47:46.682879 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.682886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.683481 (+ 595us) server_negotiation.cc:407] Connection header received
0928 23:47:46.683622 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.683630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.683833 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.683954 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:46.684352 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.685602 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.687142 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.688047 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.688158 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.688313 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.694195 (+ 5882us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.694327 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260928 23:47:46.697726 6646 tablet_service.cc:1467] Processing CreateTablet for tablet e0d7e303b2014db4ba13bff0bea900d2 (DEFAULT_TABLE table=test-table [id=e6570a7a2f4a4c008e605629a9ee8d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:46.699007 6646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0d7e303b2014db4ba13bff0bea900d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:46.699738 6997 tablet_bootstrap.cc:492] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Bootstrap starting.
I20260928 23:47:46.701114 6998 tablet_bootstrap.cc:492] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Bootstrap starting.
I20260928 23:47:46.702257 6997 tablet_bootstrap.cc:654] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:46.703377 6998 tablet_bootstrap.cc:654] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:46.703601 6997 log.cc:824] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:46.704722 6998 log.cc:824] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:46.705910 7001 tablet_bootstrap.cc:492] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: Bootstrap starting.
I20260928 23:47:46.706496 6997 tablet_bootstrap.cc:492] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: No bootstrap required, opened a new log
I20260928 23:47:46.706578 6998 tablet_bootstrap.cc:492] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: No bootstrap required, opened a new log
I20260928 23:47:46.706722 6998 ts_tablet_manager.cc:1397] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:47:46.706712 6997 ts_tablet_manager.cc:1397] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 23:47:46.708019 7001 tablet_bootstrap.cc:654] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:46.708696 7001 log.cc:824] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:46.710479 7001 tablet_bootstrap.cc:492] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: No bootstrap required, opened a new log
I20260928 23:47:46.710690 7001 ts_tablet_manager.cc:1397] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:47:46.710739 6997 raft_consensus.cc:348] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.710740 6998 raft_consensus.cc:348] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.711022 6998 raft_consensus.cc:374] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:46.711022 6997 raft_consensus.cc:374] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:46.711081 6998 raft_consensus.cc:729] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c689cbda104f1f9eb01e88b237d819, State: Initialized, Role: FOLLOWER
I20260928 23:47:46.711091 6997 raft_consensus.cc:729] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7e507065a04040b1d8d79df3759702, State: Initialized, Role: FOLLOWER
I20260928 23:47:46.711570 6997 consensus_queue.cc:260] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.711570 6998 consensus_queue.cc:260] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.712400 6980 heartbeater.cc:503] Master 127.21.112.126:46281 was elected leader, sending a full tablet report...
I20260928 23:47:46.712639 6998 ts_tablet_manager.cc:1428] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:47:46.714434 6846 heartbeater.cc:503] Master 127.21.112.126:46281 was elected leader, sending a full tablet report...
I20260928 23:47:46.714752 6997 ts_tablet_manager.cc:1428] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 23:47:46.716338 7001 raft_consensus.cc:348] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.716641 7001 raft_consensus.cc:374] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:46.716704 7001 raft_consensus.cc:729] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38172be2eae484785c5e731726ef19f, State: Initialized, Role: FOLLOWER
I20260928 23:47:46.717201 7001 consensus_queue.cc:260] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.717969 6712 heartbeater.cc:503] Master 127.21.112.126:46281 was elected leader, sending a full tablet report...
I20260928 23:47:46.719067 7001 ts_tablet_manager.cc:1428] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 23:47:46.834026 7005 raft_consensus.cc:482] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:46.834280 7005 raft_consensus.cc:504] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.835639 7005 leader_election.cc:288] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93c689cbda104f1f9eb01e88b237d819 (127.21.112.67:34783), 1e7e507065a04040b1d8d79df3759702 (127.21.112.66:32779)
I20260928 23:47:46.841336 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.836278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:55253 (local address 127.21.112.66:32779)
0928 23:47:46.836452 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.836456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.836468 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:47:46.836515 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.836517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.836566 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.836643 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:46.837068 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.837929 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.839945 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.840557 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.840682 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.840734 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.841129 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.841207 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 23:47:46.841377 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.835968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:37583 (local address 127.21.112.67:34783)
0928 23:47:46.836148 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.836156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.836240 (+ 84us) server_negotiation.cc:407] Connection header received
0928 23:47:46.836480 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.836484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.836544 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.836638 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:46.837068 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.837929 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.839945 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.840559 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.840682 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.840734 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.841129 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.841222 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 23:47:46.841419 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.835925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:32779 (local address 127.21.112.65:55253)
0928 23:47:46.836319 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.836331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.836421 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.836669 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.836672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.836711 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:46.836913 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.836921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.838072 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.838076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.839806 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.839820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.840713 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.840717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.840944 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.840986 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.841073 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.841213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":319,"spinlock_wait_cycles":6528,"thread_start_us":143,"threads_started":1}
I20260928 23:47:46.841419 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.835830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:34783 (local address 127.21.112.65:37583)
0928 23:47:46.836146 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.836161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:46.836336 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:46.836669 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:46.836672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:46.836710 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:46.836913 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.836921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.838072 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.838076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.839806 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:46.839820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.840713 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.840717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:46.840939 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.840986 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 23:47:46.841073 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:47:46.841221 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":267,"spinlock_wait_cycles":27904,"thread_start_us":120,"threads_started":1}
I20260928 23:47:46.842099 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d7e303b2014db4ba13bff0bea900d2" candidate_uuid: "e38172be2eae484785c5e731726ef19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7e507065a04040b1d8d79df3759702" is_pre_election: true
I20260928 23:47:46.842101 6934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d7e303b2014db4ba13bff0bea900d2" candidate_uuid: "e38172be2eae484785c5e731726ef19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c689cbda104f1f9eb01e88b237d819" is_pre_election: true
I20260928 23:47:46.842430 6800 raft_consensus.cc:2454] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38172be2eae484785c5e731726ef19f in term 0.
I20260928 23:47:46.842443 6934 raft_consensus.cc:2454] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38172be2eae484785c5e731726ef19f in term 0.
I20260928 23:47:46.842934 6599 leader_election.cc:302] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7e507065a04040b1d8d79df3759702, e38172be2eae484785c5e731726ef19f; no voters:
I20260928 23:47:46.843219 7005 raft_consensus.cc:2781] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:46.843305 7005 raft_consensus.cc:482] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:46.843348 7005 raft_consensus.cc:3037] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:46.844419 7005 raft_consensus.cc:504] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.844769 7005 leader_election.cc:288] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [CANDIDATE]: Term 1 election: Requested vote from peers 93c689cbda104f1f9eb01e88b237d819 (127.21.112.67:34783), 1e7e507065a04040b1d8d79df3759702 (127.21.112.66:32779)
I20260928 23:47:46.845153 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d7e303b2014db4ba13bff0bea900d2" candidate_uuid: "e38172be2eae484785c5e731726ef19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7e507065a04040b1d8d79df3759702"
I20260928 23:47:46.845151 6934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0d7e303b2014db4ba13bff0bea900d2" candidate_uuid: "e38172be2eae484785c5e731726ef19f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c689cbda104f1f9eb01e88b237d819"
I20260928 23:47:46.845265 6800 raft_consensus.cc:3037] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:46.845292 6934 raft_consensus.cc:3037] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:46.846427 6934 raft_consensus.cc:2454] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38172be2eae484785c5e731726ef19f in term 1.
I20260928 23:47:46.846463 6800 raft_consensus.cc:2454] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38172be2eae484785c5e731726ef19f in term 1.
I20260928 23:47:46.846788 6599 leader_election.cc:302] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7e507065a04040b1d8d79df3759702, e38172be2eae484785c5e731726ef19f; no voters:
I20260928 23:47:46.846997 7005 raft_consensus.cc:2781] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:46.847224 7005 raft_consensus.cc:686] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 1 LEADER]: Becoming Leader. State: Replica: e38172be2eae484785c5e731726ef19f, State: Running, Role: LEADER
I20260928 23:47:46.847581 7005 consensus_queue.cc:237] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } }
I20260928 23:47:46.851269 6517 catalog_manager.cc:5495] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f reported cstate change: term changed from 0 to 1, leader changed from <none> to e38172be2eae484785c5e731726ef19f (127.21.112.65). New cstate: current_term: 1 leader_uuid: "e38172be2eae484785c5e731726ef19f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38172be2eae484785c5e731726ef19f" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 35557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 } health_report { overall_health: UNKNOWN } } }
W20260928 23:47:46.874907 6847 tablet.cc:2369] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:47:46.881600 6981 tablet.cc:2369] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:46.882375 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.878170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.21.112.65:35557)
0928 23:47:46.878347 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:46.878351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:46.878364 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:47:46.878447 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:46.878449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:46.878501 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:46.878586 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:46.878872 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.879413 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.880697 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:46.880917 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:46.881000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:46.881339 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:46.881908 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:46.882031 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:46.882112 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:47:46.882183 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:47:46.888753 6800 raft_consensus.cc:1261] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 1 FOLLOWER]: Refusing update from remote peer e38172be2eae484785c5e731726ef19f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:46.888784 6934 raft_consensus.cc:1261] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 1 FOLLOWER]: Refusing update from remote peer e38172be2eae484785c5e731726ef19f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:46.889521 7008 consensus_queue.cc:1035] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7e507065a04040b1d8d79df3759702" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:46.889770 7005 consensus_queue.cc:1035] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c689cbda104f1f9eb01e88b237d819" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:46.908849 6517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:46.909054 6517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:46.911695 6517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6513c39beaf471d909cacf417b27daf: Sending DeleteTablet for 3 replicas of tablet e0d7e303b2014db4ba13bff0bea900d2
I20260928 23:47:46.912510 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d7e303b2014db4ba13bff0bea900d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:36476
I20260928 23:47:46.912554 6914 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d7e303b2014db4ba13bff0bea900d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:51968
I20260928 23:47:46.912563 6646 tablet_service.cc:1514] Processing DeleteTablet for tablet e0d7e303b2014db4ba13bff0bea900d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:46 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:39798
I20260928 23:47:46.913092 7023 tablet_replica.cc:330] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f: stopping tablet replica
I20260928 23:47:46.913266 7022 tablet_replica.cc:330] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: stopping tablet replica
I20260928 23:47:46.913472 7023 raft_consensus.cc:2229] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:47:46.913610 7022 raft_consensus.cc:2229] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:46.913939 7023 raft_consensus.cc:2258] T e0d7e303b2014db4ba13bff0bea900d2 P e38172be2eae484785c5e731726ef19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:46.913959 7022 raft_consensus.cc:2258] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:46.914085 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 6583
I20260928 23:47:46.914767 7021 tablet_replica.cc:330] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: stopping tablet replica
I20260928 23:47:46.915086 7021 raft_consensus.cc:2229] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:46.915357 7021 raft_consensus.cc:2258] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:46.916949 7022 ts_tablet_manager.cc:1905] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:46.917061 7021 ts_tablet_manager.cc:1905] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:46.919567 7021 ts_tablet_manager.cc:1918] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:46.919703 7021 log.cc:1196] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/e0d7e303b2014db4ba13bff0bea900d2
I20260928 23:47:46.920033 7021 ts_tablet_manager.cc:1939] T e0d7e303b2014db4ba13bff0bea900d2 P 93c689cbda104f1f9eb01e88b237d819: Deleting consensus metadata
I20260928 23:47:46.920373 7022 ts_tablet_manager.cc:1918] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:46.920476 7022 log.cc:1196] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/e0d7e303b2014db4ba13bff0bea900d2
I20260928 23:47:46.920771 7022 ts_tablet_manager.cc:1939] T e0d7e303b2014db4ba13bff0bea900d2 P 1e7e507065a04040b1d8d79df3759702: Deleting consensus metadata
I20260928 23:47:46.921351 6506 catalog_manager.cc:4841] TS 93c689cbda104f1f9eb01e88b237d819 (127.21.112.67:34783): tablet e0d7e303b2014db4ba13bff0bea900d2 (table test-table [id=e6570a7a2f4a4c008e605629a9ee8d3f]) successfully deleted
I20260928 23:47:46.921957 6505 catalog_manager.cc:4841] TS 1e7e507065a04040b1d8d79df3759702 (127.21.112.66:32779): tablet e0d7e303b2014db4ba13bff0bea900d2 (table test-table [id=e6570a7a2f4a4c008e605629a9ee8d3f]) successfully deleted
W20260928 23:47:46.922492 6507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:35557 (error 108)
I20260928 23:47:46.922713 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 6717
I20260928 23:47:46.924516 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:46.923717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:35557 (local address 127.0.0.1:39824)
0928 23:47:46.924312 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:46.924408 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486}
W20260928 23:47:46.924746 6507 catalog_manager.cc:4584] TS e38172be2eae484785c5e731726ef19f (127.21.112.65:35557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0d7e303b2014db4ba13bff0bea900d2: Network error: Client connection negotiation failed: client connection to 127.21.112.65:35557: connect: Connection refused (error 111)
I20260928 23:47:46.929430 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 6851
I20260928 23:47:46.935619 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 6489
2026-09-28T23:47:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2411 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T23:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:46Z Disabled control of system clock
I20260928 23:47:46.966861 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:38641
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:38641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:47:47.081051 7028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:47.081368 7028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:47.081413 7028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:47.085522 7028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:47.085654 7028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:47.085672 7028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:47.085690 7028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:47.085704 7028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:47.091203 7028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:38641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:38641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:47.092360 7028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:47.093834 7028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:47.103108 7036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:47.103159 7033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:47.103200 7028 server_base.cc:1034] running on GCE node
W20260928 23:47:47.103158 7034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:47.103968 7028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:47.105114 7028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:47.106328 7028 hybrid_clock.cc:648] HybridClock initialized: now 1790639267106285 us; error 57 us; skew 500 ppm
I20260928 23:47:47.108572 7028 webserver.cc:460] Webserver started at http://127.21.112.126:45283/ using document root <none> and password file <none>
I20260928 23:47:47.109193 7028 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:47.109264 7028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:47.109503 7028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:47.111619 7028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "1ff15cd183a74784b9bc57f32b908abb"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "0b8a51b202ac74e3b84a39deb5d2f426"
server_key_iv: "6aed77525345f8e96316f2279787886b"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.112185 7028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "1ff15cd183a74784b9bc57f32b908abb"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "0b8a51b202ac74e3b84a39deb5d2f426"
server_key_iv: "6aed77525345f8e96316f2279787886b"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.115937 7028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 23:47:47.118474 7042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.119690 7028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:47.119830 7028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "1ff15cd183a74784b9bc57f32b908abb"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "0b8a51b202ac74e3b84a39deb5d2f426"
server_key_iv: "6aed77525345f8e96316f2279787886b"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.119943 7028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:47.145044 7028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:47.145879 7028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:47.146088 7028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:47.153278 7028 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:38641
I20260928 23:47:47.153314 7094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:38641 every 8 connection(s)
I20260928 23:47:47.154345 7028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:47.157867 7095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:47.163491 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7028
I20260928 23:47:47.163641 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:47.163964 21953 external_mini_cluster.cc:1428] Setting key 21a07b9828865ec9926013f49ff8de0c
I20260928 23:47:47.164429 7095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb: Bootstrap starting.
I20260928 23:47:47.167137 7095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:47.167939 7095 log.cc:824] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:47.169937 7095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb: No bootstrap required, opened a new log
I20260928 23:47:47.173794 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.165585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.21.112.126:38641)
0928 23:47:47.166365 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.166379 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.166416 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:47:47.167063 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.167080 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.167614 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.167965 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:47.168400 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.169369 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.170439 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.170920 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.171099 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.171982 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:47.171987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:47.171999 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:47.172389 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:47.172444 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:47.172463 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:47.172576 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.172720 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.172984 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":164,"threads_started":1}
I20260928 23:47:47.173774 7095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff15cd183a74784b9bc57f32b908abb" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 38641 } }
I20260928 23:47:47.174064 7095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:47.174118 7095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff15cd183a74784b9bc57f32b908abb, State: Initialized, Role: FOLLOWER
I20260928 23:47:47.174602 7095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff15cd183a74784b9bc57f32b908abb" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 38641 } }
I20260928 23:47:47.174741 7095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:47.174798 7095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:47.174885 7095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:47.175845 7095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff15cd183a74784b9bc57f32b908abb" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 38641 } }
I20260928 23:47:47.176214 7095 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff15cd183a74784b9bc57f32b908abb; no voters:
I20260928 23:47:47.176563 7095 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:47.176707 7100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:47.176961 7100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [term 1 LEADER]: Becoming Leader. State: Replica: 1ff15cd183a74784b9bc57f32b908abb, State: Running, Role: LEADER
I20260928 23:47:47.177362 7100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff15cd183a74784b9bc57f32b908abb" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 38641 } }
I20260928 23:47:47.177870 7095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:47.179797 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff15cd183a74784b9bc57f32b908abb. Latest consensus state: current_term: 1 leader_uuid: "1ff15cd183a74784b9bc57f32b908abb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff15cd183a74784b9bc57f32b908abb" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 38641 } } }
I20260928 23:47:47.179947 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:47.180220 7101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ff15cd183a74784b9bc57f32b908abb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff15cd183a74784b9bc57f32b908abb" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 38641 } } }
I20260928 23:47:47.180320 7101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:47.180584 7109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:47.184079 7109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:47.190303 7109 catalog_manager.cc:1347] Generated new cluster ID: 83499f4cb321417fa8a1c100904f038d
I20260928 23:47:47.190404 7109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:47.216801 7109 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:47.218086 7109 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:47.224320 7109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb: Generated new TSK 0
I20260928 23:47:47.225020 7109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:47:47.230022 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:38641
--builtin_ntp_servers=127.21.112.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:47:47.355594 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:47.355984 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:47.356061 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:47.362612 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:47.362712 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:47.362812 7119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:47.368240 7119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:38641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:47.369747 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:47.371178 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:47.381520 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:47.381603 7125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:47.381655 7119 server_base.cc:1034] running on GCE node
W20260928 23:47:47.381595 7127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:47.382136 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:47.382783 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:47.383970 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790639267383959 us; error 35 us; skew 500 ppm
I20260928 23:47:47.385865 7119 webserver.cc:460] Webserver started at http://127.21.112.65:32799/ using document root <none> and password file <none>
I20260928 23:47:47.386518 7119 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:47.386606 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:47.386814 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:47.388890 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "e290ccf7fae64581bd2ecc31fba88642"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "2d9d6815527ab070f7e3513a1e3964c2"
server_key_iv: "2c60afd2f753796e2abf0bd7cc8e8a7c"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.389472 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "e290ccf7fae64581bd2ecc31fba88642"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "2d9d6815527ab070f7e3513a1e3964c2"
server_key_iv: "2c60afd2f753796e2abf0bd7cc8e8a7c"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.393131 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:47.395792 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.397078 7119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 23:47:47.397235 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "e290ccf7fae64581bd2ecc31fba88642"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "2d9d6815527ab070f7e3513a1e3964c2"
server_key_iv: "2c60afd2f753796e2abf0bd7cc8e8a7c"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.397411 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:47.416513 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:47.417325 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:47.417599 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:47.418252 7119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:47.419260 7119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:47.419332 7119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.419407 7119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:47.419451 7119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.429304 7119 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:34357
I20260928 23:47:47.429332 7246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:34357 every 8 connection(s)
I20260928 23:47:47.430353 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:47.437119 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7119
I20260928 23:47:47.437234 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:47:47.437603 21953 external_mini_cluster.cc:1428] Setting key 07b7423f78509a5addc97b1034134ee8
I20260928 23:47:47.439862 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:38641
--builtin_ntp_servers=127.21.112.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:47:47.440258 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.432262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:59037 (local address 127.21.112.126:38641)
0928 23:47:47.432436 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.432441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.433585 (+ 1144us) server_negotiation.cc:407] Connection header received
0928 23:47:47.434946 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.434949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.434996 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.435079 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:47.435988 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.436554 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.437935 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.438241 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.438346 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.439641 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:47.439647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:47.439653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:47.439670 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:47.439697 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:47.439710 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:47.439792 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.439962 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.440046 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:47:47.441080 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.432692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:38641 (local address 127.21.112.65:59037)
0928 23:47:47.433373 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:47.433415 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:47.434744 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:47.435245 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:47.435251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:47.435336 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:47.435830 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.435842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.436675 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.436677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.437817 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.437827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.438411 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.438416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.438601 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.439380 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:47.439382 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:47.439422 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:47.439486 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:47.439801 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:47.439808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:47.439812 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:47:47.439921 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:47:47.440243 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260928 23:47:47.442476 7247 heartbeater.cc:347] Connected to a master server at 127.21.112.126:38641
I20260928 23:47:47.442759 7247 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:47.443306 7247 heartbeater.cc:511] Master 127.21.112.126:38641 requested a full tablet report, sending...
I20260928 23:47:47.444973 7058 ts_manager.cc:194] Registered new tserver with Master: e290ccf7fae64581bd2ecc31fba88642 (127.21.112.65:34357)
I20260928 23:47:47.446224 7058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.65:59037
W20260928 23:47:47.555061 7250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:47.555343 7250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:47.555382 7250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:47.559441 7250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:47.559499 7250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:47.559554 7250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:47.564414 7250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:38641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:47.565658 7250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:47.567129 7250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:47.577207 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:47.577211 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:47.577283 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:47.577431 7250 server_base.cc:1034] running on GCE node
I20260928 23:47:47.578002 7250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:47.578717 7250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:47.579911 7250 hybrid_clock.cc:648] HybridClock initialized: now 1790639267579899 us; error 37 us; skew 500 ppm
I20260928 23:47:47.581871 7250 webserver.cc:460] Webserver started at http://127.21.112.66:35473/ using document root <none> and password file <none>
I20260928 23:47:47.582446 7250 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:47.582537 7250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:47.582774 7250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:47.584786 7250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "d68c53a1b64643bb8dc4da1cc2ce1143"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "27fa79bc88c1e9be47247070923742a1"
server_key_iv: "0ebcfb0809e3c52e1dbf20c6b8f0b8a1"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.585371 7250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "d68c53a1b64643bb8dc4da1cc2ce1143"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "27fa79bc88c1e9be47247070923742a1"
server_key_iv: "0ebcfb0809e3c52e1dbf20c6b8f0b8a1"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.589176 7250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:47.591740 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.593220 7250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:47.593385 7250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "d68c53a1b64643bb8dc4da1cc2ce1143"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "27fa79bc88c1e9be47247070923742a1"
server_key_iv: "0ebcfb0809e3c52e1dbf20c6b8f0b8a1"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.593518 7250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:47.605804 7250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:47.606546 7250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:47.606766 7250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:47.607429 7250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:47.608445 7250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:47.608525 7250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.608605 7250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:47.608654 7250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.618636 7250 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:39397
I20260928 23:47:47.618656 7377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:39397 every 8 connection(s)
I20260928 23:47:47.619621 7250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:47.626917 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7250
I20260928 23:47:47.627041 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:47:47.627368 21953 external_mini_cluster.cc:1428] Setting key 0dd05396a2ebc3946d0e5a5ab81d688b
I20260928 23:47:47.629490 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:38641
--builtin_ntp_servers=127.21.112.84:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:47:47.630923 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.622506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:51429 (local address 127.21.112.126:38641)
0928 23:47:47.622667 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.622678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.623732 (+ 1054us) server_negotiation.cc:407] Connection header received
0928 23:47:47.625294 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.625297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.625353 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.625433 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:47.626391 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.627183 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.628675 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.628977 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.629084 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.630322 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:47.630329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:47.630335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:47.630353 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:47.630381 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:47.630394 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:47.630473 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.630622 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.630696 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:47:47.631765 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.622881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:38641 (local address 127.21.112.66:51429)
0928 23:47:47.623569 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:47.623609 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:47.625081 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:47.625668 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:47.625675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:47.625760 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:47.626246 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.626257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.627335 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.627338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.628549 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.628560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.629129 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.629132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.629287 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.630029 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:47.630031 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:47.630083 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:47.630154 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:47.630479 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:47.630486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:47.630490 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:47:47.630590 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 23:47:47.630837 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260928 23:47:47.633015 7378 heartbeater.cc:347] Connected to a master server at 127.21.112.126:38641
I20260928 23:47:47.633281 7378 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:47.633921 7378 heartbeater.cc:511] Master 127.21.112.126:38641 requested a full tablet report, sending...
I20260928 23:47:47.635002 7058 ts_manager.cc:194] Registered new tserver with Master: d68c53a1b64643bb8dc4da1cc2ce1143 (127.21.112.66:39397)
I20260928 23:47:47.635591 7058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.66:51429
W20260928 23:47:47.748193 7381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:47.748502 7381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:47.748540 7381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:47.752835 7381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:47.752914 7381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:47.752975 7381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:47.758208 7381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:38641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:47.759462 7381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:47.760950 7381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:47.771155 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:47.771189 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:47.771166 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:47.771436 7381 server_base.cc:1034] running on GCE node
I20260928 23:47:47.772024 7381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:47.772732 7381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:47.773931 7381 hybrid_clock.cc:648] HybridClock initialized: now 1790639267773903 us; error 53 us; skew 500 ppm
I20260928 23:47:47.776022 7381 webserver.cc:460] Webserver started at http://127.21.112.67:44197/ using document root <none> and password file <none>
I20260928 23:47:47.776641 7381 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:47.776746 7381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:47.777000 7381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:47.779368 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "31219aff9e5a4910a42cbf33371512c0"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "0873ca464ed99470f1702854ab1a9ced"
server_key_iv: "ca2ad6ebb04afbfc9e1073419ab907fa"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.779970 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "31219aff9e5a4910a42cbf33371512c0"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "0873ca464ed99470f1702854ab1a9ced"
server_key_iv: "ca2ad6ebb04afbfc9e1073419ab907fa"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.784020 7381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 23:47:47.786895 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.788363 7381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:47:47.788537 7381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "31219aff9e5a4910a42cbf33371512c0"
format_stamp: "Formatted at 2026-09-28 23:47:47 on dist-test-slave-c3s8"
server_key: "0873ca464ed99470f1702854ab1a9ced"
server_key_iv: "ca2ad6ebb04afbfc9e1073419ab907fa"
server_key_version: "encryptionkey@0"
I20260928 23:47:47.788674 7381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:47.817960 7381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:47.818799 7381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:47.819054 7381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:47.819737 7381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:47.820811 7381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:47.820894 7381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.820972 7381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:47.821030 7381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:47.831919 7381 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:45299
I20260928 23:47:47.831938 7508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:45299 every 8 connection(s)
I20260928 23:47:47.832947 7381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:47.837366 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7381
I20260928 23:47:47.837508 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:47:47.837888 21953 external_mini_cluster.cc:1428] Setting key 2259e06c64f3be5adb5a027e8130b6c7
I20260928 23:47:47.843290 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.834976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:52155 (local address 127.21.112.126:38641)
0928 23:47:47.835156 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.835162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.836293 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 23:47:47.837825 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.837829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.837892 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.837999 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:47.839312 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.840028 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.841232 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.841511 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.841676 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.842846 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:47.842849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:47.842852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:47.842864 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:47.842884 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:47.842892 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:47.842962 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.843109 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.843162 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:47:47.844204 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.835456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:38641 (local address 127.21.112.67:52155)
0928 23:47:47.836131 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:47.836172 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:47.837532 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:47.838314 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:47.838327 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:47.838451 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:47.839149 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.839170 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.840197 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.840200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.841105 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.841115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.841822 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.841825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.841975 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.842566 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:47.842567 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:47.842607 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:47.842670 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:47.842974 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:47.842981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:47.842984 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:47:47.843089 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:47:47.843362 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260928 23:47:47.845870 7509 heartbeater.cc:347] Connected to a master server at 127.21.112.126:38641
I20260928 23:47:47.846087 7509 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:47.846599 7509 heartbeater.cc:511] Master 127.21.112.126:38641 requested a full tablet report, sending...
I20260928 23:47:47.847728 7058 ts_manager.cc:194] Registered new tserver with Master: 31219aff9e5a4910a42cbf33371512c0 (127.21.112.67:45299)
I20260928 23:47:47.848318 7058 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.67:52155
I20260928 23:47:47.852092 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:47.858300 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.854192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.21.112.126:38641)
0928 23:47:47.854312 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.854316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.854601 (+ 285us) server_negotiation.cc:407] Connection header received
0928 23:47:47.854752 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.854754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.854797 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.854915 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:47.855209 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.855719 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.856568 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.856780 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.856884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.857807 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:47.857811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:47.857813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:47.857826 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:47.857843 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:47.857851 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:47.857965 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.858060 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.858119 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:47:47.861320 7058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:47.864408 7058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:47.880187 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.873752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45299 (local address 127.0.0.1:50128)
0928 23:47:47.874749 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:47.874763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:47.874850 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:47.875649 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:47.875657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:47.875681 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:47.875849 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.875857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.877297 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.877301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.878557 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.878567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.879636 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.879640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.879752 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.879796 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:47:47.879896 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 23:47:47.879992 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":73,"thread_start_us":107,"threads_started":1}
I20260928 23:47:47.880298 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.874460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.21.112.67:45299)
0928 23:47:47.875105 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.875115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.875133 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:47:47.875196 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.875200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.875438 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.875648 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:47.875974 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.877181 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.878671 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.879489 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.879796 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.879977 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.880032 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.880127 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":156,"threads_started":1}
I20260928 23:47:47.880367 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.873764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.21.112.66:39397)
0928 23:47:47.874956 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.874969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.874996 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:47:47.875075 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.875080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.875293 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.875701 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:47.876172 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.877287 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.878796 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.879507 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.879608 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.879796 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.880096 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.880195 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":104,"threads_started":1}
I20260928 23:47:47.880904 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.874418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:34357 (local address 127.0.0.1:34764)
0928 23:47:47.875119 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:47.875134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:47.875225 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:47.875856 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:47.875858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:47.875879 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:47.876045 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.876052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.877051 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.877056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.878684 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.878688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.879588 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.879591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.879697 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.880615 (+ 918us) client_negotiation.cc:769] Sending connection context
0928 23:47:47.880700 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:47:47.880775 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":87,"spinlock_wait_cycles":2176,"thread_start_us":103,"threads_started":1}
I20260928 23:47:47.881026 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.874633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.21.112.65:34357)
0928 23:47:47.874926 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:47.874939 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:47.875330 (+ 391us) server_negotiation.cc:407] Connection header received
0928 23:47:47.875433 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:47.875440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:47.875650 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:47.875798 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:47.876202 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.876935 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.878782 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.879464 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.879590 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.879807 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:47.880748 (+ 941us) server_negotiation.cc:299] Negotiation successful
0928 23:47:47.880852 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260928 23:47:47.881484 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:47.873655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:39397 (local address 127.0.0.1:57356)
0928 23:47:47.874270 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:47.874325 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:47.874451 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:47.875843 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:47.875846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:47.875865 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:47.876049 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.876055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.877423 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.877426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.878659 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:47.878670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:47.879857 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:47.879861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:47.879941 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:47.879963 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:47:47.880048 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:47:47.880117 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":64,"thread_start_us":99,"threads_started":1}
I20260928 23:47:47.883642 7181 tablet_service.cc:1467] Processing CreateTablet for tablet 6e770ea84bda4741a18347e2987c83af (DEFAULT_TABLE table=test-table [id=456ad2a5b0014ed98cfe9c6d3ed96ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:47.883673 7443 tablet_service.cc:1467] Processing CreateTablet for tablet 6e770ea84bda4741a18347e2987c83af (DEFAULT_TABLE table=test-table [id=456ad2a5b0014ed98cfe9c6d3ed96ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:47.883922 7312 tablet_service.cc:1467] Processing CreateTablet for tablet 6e770ea84bda4741a18347e2987c83af (DEFAULT_TABLE table=test-table [id=456ad2a5b0014ed98cfe9c6d3ed96ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:47.884570 7181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e770ea84bda4741a18347e2987c83af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:47.884702 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e770ea84bda4741a18347e2987c83af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:47.884729 7312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e770ea84bda4741a18347e2987c83af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:47.891042 7526 tablet_bootstrap.cc:492] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: Bootstrap starting.
I20260928 23:47:47.892587 7528 tablet_bootstrap.cc:492] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Bootstrap starting.
I20260928 23:47:47.893349 7526 tablet_bootstrap.cc:654] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:47.893885 7527 tablet_bootstrap.cc:492] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Bootstrap starting.
I20260928 23:47:47.894258 7526 log.cc:824] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:47.894784 7528 tablet_bootstrap.cc:654] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:47.895587 7528 log.cc:824] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:47.895982 7527 tablet_bootstrap.cc:654] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:47.896652 7527 log.cc:824] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:47.896852 7526 tablet_bootstrap.cc:492] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: No bootstrap required, opened a new log
I20260928 23:47:47.897085 7526 ts_tablet_manager.cc:1397] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 23:47:47.898648 7527 tablet_bootstrap.cc:492] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: No bootstrap required, opened a new log
I20260928 23:47:47.898861 7527 ts_tablet_manager.cc:1397] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 23:47:47.900704 7526 raft_consensus.cc:348] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:47.900935 7526 raft_consensus.cc:374] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:47.900993 7526 raft_consensus.cc:729] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e290ccf7fae64581bd2ecc31fba88642, State: Initialized, Role: FOLLOWER
I20260928 23:47:47.901451 7526 consensus_queue.cc:260] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:47.902127 7247 heartbeater.cc:503] Master 127.21.112.126:38641 was elected leader, sending a full tablet report...
I20260928 23:47:47.902539 7526 ts_tablet_manager.cc:1428] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:47:47.902894 7527 raft_consensus.cc:348] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:47.903137 7527 raft_consensus.cc:374] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:47.903196 7527 raft_consensus.cc:729] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68c53a1b64643bb8dc4da1cc2ce1143, State: Initialized, Role: FOLLOWER
I20260928 23:47:47.903687 7527 consensus_queue.cc:260] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:47.904366 7378 heartbeater.cc:503] Master 127.21.112.126:38641 was elected leader, sending a full tablet report...
I20260928 23:47:47.904431 7528 tablet_bootstrap.cc:492] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: No bootstrap required, opened a new log
I20260928 23:47:47.904613 7528 ts_tablet_manager.cc:1397] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260928 23:47:47.904678 7527 ts_tablet_manager.cc:1428] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260928 23:47:47.908774 7528 raft_consensus.cc:348] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:47.909021 7528 raft_consensus.cc:374] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:47.909078 7528 raft_consensus.cc:729] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31219aff9e5a4910a42cbf33371512c0, State: Initialized, Role: FOLLOWER
I20260928 23:47:47.909502 7528 consensus_queue.cc:260] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:47.910225 7509 heartbeater.cc:503] Master 127.21.112.126:38641 was elected leader, sending a full tablet report...
I20260928 23:47:47.910486 7528 ts_tablet_manager.cc:1428] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 23:47:47.931556 7248 tablet.cc:2369] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:47:48.084091 7510 tablet.cc:2369] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:47:48.120965 7379 tablet.cc:2369] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:48.169449 7533 raft_consensus.cc:482] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:48.169741 7533 raft_consensus.cc:504] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:48.170969 7533 leader_election.cc:288] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31219aff9e5a4910a42cbf33371512c0 (127.21.112.67:45299), e290ccf7fae64581bd2ecc31fba88642 (127.21.112.65:34357)
I20260928 23:47:48.175397 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:48.171378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:38743 (local address 127.21.112.65:34357)
0928 23:47:48.171505 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:48.171509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:48.171686 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:47:48.171851 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:48.171854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:48.171909 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:48.171994 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:48.172264 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.172778 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.173922 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.174539 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.174681 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:48.174745 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:48.175196 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 23:47:48.175270 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:47:48.175439 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:48.171237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45299 (local address 127.21.112.66:38285)
0928 23:47:48.171691 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:48.171708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:48.171859 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:48.172111 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:48.172113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:48.172129 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:48.172256 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:48.172263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.173000 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.173005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:48.173874 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:48.173887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.174812 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.174816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:48.175031 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:48.175086 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 23:47:48.175161 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:47:48.175259 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":137,"thread_start_us":72,"threads_started":1}
I20260928 23:47:48.175474 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:48.171354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:38285 (local address 127.21.112.67:45299)
0928 23:47:48.171491 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:48.171498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:48.171812 (+ 314us) server_negotiation.cc:407] Connection header received
0928 23:47:48.171962 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:48.171965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:48.172015 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:48.172098 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:48.172377 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.172888 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.173987 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.174594 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.174700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:48.174744 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:48.175197 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 23:47:48.175356 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:47:48.175638 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:48.171238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:34357 (local address 127.21.112.66:38743)
0928 23:47:48.171589 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:48.171602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:48.171729 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:48.172008 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:48.172011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:48.172026 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:48.172150 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:48.172156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.172892 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.172898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:48.173799 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:48.173808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.174812 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.174816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:48.175005 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:48.175079 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 23:47:48.175161 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:47:48.175256 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":83,"thread_start_us":100,"threads_started":1}
I20260928 23:47:48.176087 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770ea84bda4741a18347e2987c83af" candidate_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31219aff9e5a4910a42cbf33371512c0" is_pre_election: true
I20260928 23:47:48.176388 7201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770ea84bda4741a18347e2987c83af" candidate_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e290ccf7fae64581bd2ecc31fba88642" is_pre_election: true
I20260928 23:47:48.176414 7463 raft_consensus.cc:2454] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d68c53a1b64643bb8dc4da1cc2ce1143 in term 0.
I20260928 23:47:48.176704 7201 raft_consensus.cc:2454] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d68c53a1b64643bb8dc4da1cc2ce1143 in term 0.
I20260928 23:47:48.176981 7267 leader_election.cc:302] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31219aff9e5a4910a42cbf33371512c0, d68c53a1b64643bb8dc4da1cc2ce1143; no voters:
I20260928 23:47:48.177278 7533 raft_consensus.cc:2781] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:48.177366 7533 raft_consensus.cc:482] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:48.177409 7533 raft_consensus.cc:3037] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:48.178582 7533 raft_consensus.cc:504] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:48.178933 7533 leader_election.cc:288] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [CANDIDATE]: Term 1 election: Requested vote from peers 31219aff9e5a4910a42cbf33371512c0 (127.21.112.67:45299), e290ccf7fae64581bd2ecc31fba88642 (127.21.112.65:34357)
I20260928 23:47:48.179322 7463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770ea84bda4741a18347e2987c83af" candidate_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31219aff9e5a4910a42cbf33371512c0"
I20260928 23:47:48.179391 7201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e770ea84bda4741a18347e2987c83af" candidate_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e290ccf7fae64581bd2ecc31fba88642"
I20260928 23:47:48.179481 7463 raft_consensus.cc:3037] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:48.179541 7201 raft_consensus.cc:3037] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:48.180596 7463 raft_consensus.cc:2454] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68c53a1b64643bb8dc4da1cc2ce1143 in term 1.
I20260928 23:47:48.180780 7201 raft_consensus.cc:2454] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d68c53a1b64643bb8dc4da1cc2ce1143 in term 1.
I20260928 23:47:48.180926 7267 leader_election.cc:302] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31219aff9e5a4910a42cbf33371512c0, d68c53a1b64643bb8dc4da1cc2ce1143; no voters:
I20260928 23:47:48.181119 7533 raft_consensus.cc:2781] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:48.181324 7533 raft_consensus.cc:686] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 1 LEADER]: Becoming Leader. State: Replica: d68c53a1b64643bb8dc4da1cc2ce1143, State: Running, Role: LEADER
I20260928 23:47:48.181679 7533 consensus_queue.cc:237] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } }
I20260928 23:47:48.184795 7054 catalog_manager.cc:5495] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 reported cstate change: term changed from 0 to 1, leader changed from <none> to d68c53a1b64643bb8dc4da1cc2ce1143 (127.21.112.66). New cstate: current_term: 1 leader_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68c53a1b64643bb8dc4da1cc2ce1143" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 } health_report { overall_health: UNKNOWN } } }
I20260928 23:47:48.244400 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:48.240566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.21.112.66:39397)
0928 23:47:48.240705 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:48.240709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:48.240752 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:47:48.240829 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:48.240831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:48.240890 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:48.240977 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:48.241389 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.241955 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.242910 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:48.243114 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:48.243190 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:48.243602 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:48.244040 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:48.244142 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:48.244236 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 23:47:48.244288 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:47:48.250401 7201 raft_consensus.cc:1261] T 6e770ea84bda4741a18347e2987c83af P e290ccf7fae64581bd2ecc31fba88642 [term 1 FOLLOWER]: Refusing update from remote peer d68c53a1b64643bb8dc4da1cc2ce1143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:48.250401 7463 raft_consensus.cc:1261] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 1 FOLLOWER]: Refusing update from remote peer d68c53a1b64643bb8dc4da1cc2ce1143: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:48.251164 7537 consensus_queue.cc:1035] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e290ccf7fae64581bd2ecc31fba88642" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 34357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:48.251348 7533 consensus_queue.cc:1035] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31219aff9e5a4910a42cbf33371512c0" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:48.261732 7541 mvcc.cc:205] Tried to move back new op lower bound from 7334458442746105856 to 7334458442474815488. Current Snapshot: MvccSnapshot[applied={T|T < 7334458442746105856 or (T in {7334458442746105856})}]
I20260928 23:47:48.268110 7058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:48.268265 7058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:48.270696 7058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ff15cd183a74784b9bc57f32b908abb: Sending DeleteTablet for 3 replicas of tablet 6e770ea84bda4741a18347e2987c83af
I20260928 23:47:48.271554 7312 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e770ea84bda4741a18347e2987c83af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:48 UTC) from {username='slave'} at 127.0.0.1:57356
I20260928 23:47:48.271785 7181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e770ea84bda4741a18347e2987c83af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:48 UTC) from {username='slave'} at 127.0.0.1:34764
I20260928 23:47:48.272073 7443 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e770ea84bda4741a18347e2987c83af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:48 UTC) from {username='slave'} at 127.0.0.1:50128
I20260928 23:47:48.272169 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7119
I20260928 23:47:48.272589 7549 tablet_replica.cc:330] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: stopping tablet replica
I20260928 23:47:48.272977 7549 raft_consensus.cc:2229] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:47:48.273356 7549 raft_consensus.cc:2258] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:48.274413 7551 tablet_replica.cc:330] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: stopping tablet replica
I20260928 23:47:48.274747 7551 raft_consensus.cc:2229] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:48.275060 7551 raft_consensus.cc:2258] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:48.275070 7549 ts_tablet_manager.cc:1905] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:48.276597 7551 ts_tablet_manager.cc:1905] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:48.278208 7549 ts_tablet_manager.cc:1918] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:48.278304 7549 log.cc:1196] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/6e770ea84bda4741a18347e2987c83af
I20260928 23:47:48.278604 7549 ts_tablet_manager.cc:1939] T 6e770ea84bda4741a18347e2987c83af P d68c53a1b64643bb8dc4da1cc2ce1143: Deleting consensus metadata
I20260928 23:47:48.279078 7551 ts_tablet_manager.cc:1918] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:48.279227 7551 log.cc:1196] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/6e770ea84bda4741a18347e2987c83af
I20260928 23:47:48.279561 7551 ts_tablet_manager.cc:1939] T 6e770ea84bda4741a18347e2987c83af P 31219aff9e5a4910a42cbf33371512c0: Deleting consensus metadata
I20260928 23:47:48.279917 7043 catalog_manager.cc:4841] TS d68c53a1b64643bb8dc4da1cc2ce1143 (127.21.112.66:39397): tablet 6e770ea84bda4741a18347e2987c83af (table test-table [id=456ad2a5b0014ed98cfe9c6d3ed96ef3]) successfully deleted
I20260928 23:47:48.280684 7045 catalog_manager.cc:4841] TS 31219aff9e5a4910a42cbf33371512c0 (127.21.112.67:45299): tablet 6e770ea84bda4741a18347e2987c83af (table test-table [id=456ad2a5b0014ed98cfe9c6d3ed96ef3]) successfully deleted
W20260928 23:47:48.285099 7044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:34357 (error 108)
I20260928 23:47:48.285265 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7250
I20260928 23:47:48.287537 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:48.287161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:34357 (local address 127.0.0.1:34778)
0928 23:47:48.287339 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:48.287428 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:34357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 23:47:48.287734 7044 catalog_manager.cc:4584] TS e290ccf7fae64581bd2ecc31fba88642 (127.21.112.65:34357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e770ea84bda4741a18347e2987c83af: Network error: Client connection negotiation failed: client connection to 127.21.112.65:34357: connect: Connection refused (error 111)
I20260928 23:47:48.291404 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7381
I20260928 23:47:48.297406 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7028
2026-09-28T23:47:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1360 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:48 dist-test-slave-c3s8 krb5kdc[7555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:48 dist-test-slave-c3s8 krb5kdc[7555](info): set up 2 sockets
Sep 28 23:47:48 dist-test-slave-c3s8 krb5kdc[7555](info): commencing operation
krb5kdc: starting...
W20260928 23:47:49.337926 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:47:49 dist-test-slave-c3s8 krb5kdc[7555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:47:49.465366 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46015
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:49.578828 7568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:49.579144 7568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:49.579226 7568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:49.583282 7568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:49.583382 7568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:49.583422 7568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:49.583459 7568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:49.583493 7568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:49.589094 7568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:49.590323 7568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:49.591449 7568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:49.600581 7576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:49.600534 7573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:49.600534 7574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:49.601158 7568 server_base.cc:1034] running on GCE node
I20260928 23:47:49.601819 7568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:49.602943 7568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:49.604128 7568 hybrid_clock.cc:648] HybridClock initialized: now 1790639269604094 us; error 47 us; skew 500 ppm
Sep 28 23:47:49 dist-test-slave-c3s8 krb5kdc[7555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639269, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:49.613893 7568 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:47:49.615126 7568 webserver.cc:460] Webserver started at http://127.21.112.126:44293/ using document root <none> and password file <none>
I20260928 23:47:49.615752 7568 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:49.615849 7568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:49.616093 7568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:49.618342 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b"
format_stamp: "Formatted at 2026-09-28 23:47:49 on dist-test-slave-c3s8"
server_key: "d557a22f3cffed1cea01003bd2a8e304"
server_key_iv: "d9e8b10b5250f72a44837de80b8ede7f"
server_key_version: "encryptionkey@0"
I20260928 23:47:49.618971 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b"
format_stamp: "Formatted at 2026-09-28 23:47:49 on dist-test-slave-c3s8"
server_key: "d557a22f3cffed1cea01003bd2a8e304"
server_key_iv: "d9e8b10b5250f72a44837de80b8ede7f"
server_key_version: "encryptionkey@0"
I20260928 23:47:49.622853 7568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:47:49.625671 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:49.626961 7568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:47:49.627111 7568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b"
format_stamp: "Formatted at 2026-09-28 23:47:49 on dist-test-slave-c3s8"
server_key: "d557a22f3cffed1cea01003bd2a8e304"
server_key_iv: "d9e8b10b5250f72a44837de80b8ede7f"
server_key_version: "encryptionkey@0"
I20260928 23:47:49.627233 7568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:49.650020 7568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:49.653470 7568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:49.653715 7568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:49.661259 7568 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:46015
I20260928 23:47:49.661293 7635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:46015 every 8 connection(s)
I20260928 23:47:49.662267 7568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:49.665187 7636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:49.671060 7636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b: Bootstrap starting.
I20260928 23:47:49.672537 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7568
I20260928 23:47:49.672706 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:49.673022 21953 external_mini_cluster.cc:1428] Setting key ff7d880516d5c736c02b2a11f882c92e
I20260928 23:47:49.674146 7636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:49.675052 7636 log.cc:824] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:49.677054 7636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b: No bootstrap required, opened a new log
I20260928 23:47:49.681789 7636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46015 } }
I20260928 23:47:49.682049 7636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:49.682138 7636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ece5d02e9f84c12a900c9edf9cb9a8b, State: Initialized, Role: FOLLOWER
I20260928 23:47:49.682704 7636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46015 } }
I20260928 23:47:49.682893 7636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:49.682981 7636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:49.683107 7636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 0 FOLLOWER]: Advancing to term 1
Sep 28 23:47:49 dist-test-slave-c3s8 krb5kdc[7555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:49.684293 7636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46015 } }
I20260928 23:47:49.684659 7636 leader_election.cc:302] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ece5d02e9f84c12a900c9edf9cb9a8b; no voters:
I20260928 23:47:49.684993 7636 leader_election.cc:288] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:49.685370 7641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:49.685747 7641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [term 1 LEADER]: Becoming Leader. State: Replica: 6ece5d02e9f84c12a900c9edf9cb9a8b, State: Running, Role: LEADER
I20260928 23:47:49.686132 7641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46015 } }
I20260928 23:47:49.686434 7636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:49.687866 7643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ece5d02e9f84c12a900c9edf9cb9a8b. Latest consensus state: current_term: 1 leader_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46015 } } }
I20260928 23:47:49.687997 7643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:49.688403 7642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ece5d02e9f84c12a900c9edf9cb9a8b" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46015 } } }
I20260928 23:47:49.688510 7642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:49.688855 7650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:49.690326 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:49.674335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.21.112.126:46015)
0928 23:47:49.674829 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:49.674843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:49.674885 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:47:49.675721 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:49.675747 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:49.676100 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:49.676451 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:49.677293 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:49.678954 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:49.679837 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:49.680737 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:49.680918 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:49.683957 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:49.683962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:49.683974 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:49.684001 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:49.686287 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:49.686829 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:49.686834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:49.686840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:49.686988 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:49.687367 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:49.687373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:49.687376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:49.687976 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:49.688225 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:49.689052 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 23:47:49.689427 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":141,"threads_started":1}
I20260928 23:47:49.692545 7650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:49.698473 7650 catalog_manager.cc:1347] Generated new cluster ID: ec2581cd404549f59a33dbf2994b90c1
I20260928 23:47:49.698556 7650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:49.722246 7650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:49.723637 7650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:49.733260 7650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b: Generated new TSK 0
I20260928 23:47:49.734144 7650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:47:49.773878 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46015
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:49.888020 7662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:49.888355 7662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:49.888404 7662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:49.892648 7662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:49.892717 7662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:49.892820 7662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:49.898437 7662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:49.899739 7662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:49.900846 7662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:49.911208 7670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:49.911320 7667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:49.911433 7662 server_base.cc:1034] running on GCE node
W20260928 23:47:49.911612 7668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:49.912002 7662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:49.912664 7662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:49.913857 7662 hybrid_clock.cc:648] HybridClock initialized: now 1790639269913842 us; error 27 us; skew 500 ppm
Sep 28 23:47:49 dist-test-slave-c3s8 krb5kdc[7555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639269, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:49.916776 7662 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:47:49.917759 7662 webserver.cc:460] Webserver started at http://127.21.112.65:46311/ using document root <none> and password file <none>
I20260928 23:47:49.918392 7662 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:49.918493 7662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:49.918731 7662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:49.920779 7662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "c8db28ba723143ef83261a1df0b20f14"
format_stamp: "Formatted at 2026-09-28 23:47:49 on dist-test-slave-c3s8"
server_key: "abf0dd928d85c527dab3f526a762d32a"
server_key_iv: "8dbf4959ed09c91a8611fd3bc392e6b9"
server_key_version: "encryptionkey@0"
I20260928 23:47:49.921350 7662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "c8db28ba723143ef83261a1df0b20f14"
format_stamp: "Formatted at 2026-09-28 23:47:49 on dist-test-slave-c3s8"
server_key: "abf0dd928d85c527dab3f526a762d32a"
server_key_iv: "8dbf4959ed09c91a8611fd3bc392e6b9"
server_key_version: "encryptionkey@0"
I20260928 23:47:49.925163 7662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 23:47:49.928107 7677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:49.929641 7662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:49.929776 7662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "c8db28ba723143ef83261a1df0b20f14"
format_stamp: "Formatted at 2026-09-28 23:47:49 on dist-test-slave-c3s8"
server_key: "abf0dd928d85c527dab3f526a762d32a"
server_key_iv: "8dbf4959ed09c91a8611fd3bc392e6b9"
server_key_version: "encryptionkey@0"
I20260928 23:47:49.929900 7662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:49.945693 7662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:49.949069 7662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:49.949283 7662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:49.949950 7662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:49.950809 7662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:49.950876 7662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:49.950942 7662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:49.950994 7662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:49.960232 7662 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:40345
I20260928 23:47:49.960397 7790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:40345 every 8 connection(s)
I20260928 23:47:49.961195 7662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:49.970793 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7662
I20260928 23:47:49.970983 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:47:49.971378 21953 external_mini_cluster.cc:1428] Setting key 81daf7b8a7afef0df099df0c8d48f900
Sep 28 23:47:49 dist-test-slave-c3s8 krb5kdc[7555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639269, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:49.976367 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:49.963265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:42895 (local address 127.21.112.126:46015)
0928 23:47:49.963440 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:49.963445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:49.964301 (+ 856us) server_negotiation.cc:407] Connection header received
0928 23:47:49.965669 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:49.965672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:49.965720 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:49.965789 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:49.967010 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:49.967700 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:49.968949 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:49.969191 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:49.969290 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:49.972531 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:49.972535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:49.972537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:49.972562 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:49.974221 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:49.974780 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:49.974784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:49.974785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:49.974893 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:49.975258 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:49.975263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:49.975265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:49.975452 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:49.975561 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:49.976076 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 23:47:49.976203 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:47:49.977403 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:49.963656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46015 (local address 127.21.112.65:42895)
0928 23:47:49.964124 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:49.964169 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:49.965393 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:49.965960 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:49.965967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:49.966320 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:49.966840 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:49.966853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:49.967855 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:49.967858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:49.968806 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:49.968817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:49.969428 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:49.969431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:49.969680 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:49.970416 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:49.970433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:49.972365 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:49.974348 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:49.974354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:49.974371 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:49.974645 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:49.974992 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:49.974996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:49.974997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:49.975151 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:49.975561 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:49.975568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:49.975836 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 23:47:49.976040 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 23:47:49.976397 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260928 23:47:49.978937 7791 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46015
I20260928 23:47:49.979194 7791 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:49.979720 7791 heartbeater.cc:511] Master 127.21.112.126:46015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:47:49.981307 7600 ts_manager.cc:194] Registered new tserver with Master: c8db28ba723143ef83261a1df0b20f14 (127.21.112.65:40345)
I20260928 23:47:49.982578 7600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:42895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:47:50.001703 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46015
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:50.123860 7796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:50.124152 7796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:50.124215 7796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:50.128834 7796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:50.128899 7796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:50.128962 7796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:50.134682 7796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:50.135797 7796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:50.136847 7796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:50.146538 7804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:50.146607 7802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:50.146747 7801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:50.147040 7796 server_base.cc:1034] running on GCE node
I20260928 23:47:50.147454 7796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:50.148124 7796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:50.149315 7796 hybrid_clock.cc:648] HybridClock initialized: now 1790639270149261 us; error 67 us; skew 500 ppm
Sep 28 23:47:50 dist-test-slave-c3s8 krb5kdc[7555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639270, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:50.152392 7796 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:47:50.153389 7796 webserver.cc:460] Webserver started at http://127.21.112.66:36991/ using document root <none> and password file <none>
I20260928 23:47:50.154058 7796 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:50.154156 7796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:50.154413 7796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:50.156551 7796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "93b501a3af4b450ebf6e3b537ac3a989"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "4bd742b15748fe4e3c9cb98aca608cfd"
server_key_iv: "5e5413b0828b92d2784050d04b0ad7f4"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.157127 7796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "93b501a3af4b450ebf6e3b537ac3a989"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "4bd742b15748fe4e3c9cb98aca608cfd"
server_key_iv: "5e5413b0828b92d2784050d04b0ad7f4"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.160981 7796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:47:50.163561 7811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.164881 7796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:50.165118 7796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "93b501a3af4b450ebf6e3b537ac3a989"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "4bd742b15748fe4e3c9cb98aca608cfd"
server_key_iv: "5e5413b0828b92d2784050d04b0ad7f4"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.165237 7796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:50.187556 7796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:50.190812 7796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:50.191012 7796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:50.191593 7796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:50.192405 7796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:50.192473 7796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.192545 7796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:50.192597 7796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.201647 7796 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:34523
I20260928 23:47:50.201671 7924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:34523 every 8 connection(s)
I20260928 23:47:50.202512 7796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:50.208797 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7796
I20260928 23:47:50.208948 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:47:50.209332 21953 external_mini_cluster.cc:1428] Setting key 61fd689b7d62d46416b693a0e04aa6d7
Sep 28 23:47:50 dist-test-slave-c3s8 krb5kdc[7555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639270, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:50.217842 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.204322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:35091 (local address 127.21.112.126:46015)
0928 23:47:50.204453 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.204458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.205218 (+ 760us) server_negotiation.cc:407] Connection header received
0928 23:47:50.206627 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.206629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.206678 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.206764 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:50.207937 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.208452 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.210141 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.210380 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.210500 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.213310 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:50.213315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:50.213320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:50.213354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:50.215304 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:50.216033 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:50.216040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:50.216045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:50.216138 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:50.216503 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:50.216508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:50.216513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:50.216789 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:50.216908 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.217453 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.217637 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:47:50.218681 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.204600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46015 (local address 127.21.112.66:35091)
0928 23:47:50.205050 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.205092 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.206404 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.206949 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.206956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.207308 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:50.207790 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.207804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.208616 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.208623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.209997 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.210007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.210542 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.210545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.210733 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.211399 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:50.211426 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:50.213127 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:50.215477 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:50.215486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:50.215504 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:50.215929 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:50.216258 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:50.216260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:50.216262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:50.216402 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:50.216911 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:50.216916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:50.217162 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.217362 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.217698 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:47:50.220036 7925 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46015
I20260928 23:47:50.220306 7925 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:50.220768 7925 heartbeater.cc:511] Master 127.21.112.126:46015 requested a full tablet report, sending...
I20260928 23:47:50.221918 7600 ts_manager.cc:194] Registered new tserver with Master: 93b501a3af4b450ebf6e3b537ac3a989 (127.21.112.66:34523)
I20260928 23:47:50.222537 7600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:35091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:47:50.238765 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46015
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:50.351285 7930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:50.351608 7930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:50.351716 7930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:50.355926 7930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:50.356019 7930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:50.356107 7930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:50.361611 7930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:50.362802 7930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:50.363909 7930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:50.373920 7936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:50.373944 7935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:50.374035 7930 server_base.cc:1034] running on GCE node
W20260928 23:47:50.373920 7938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:50.374631 7930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:50.375219 7930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:50.376385 7930 hybrid_clock.cc:648] HybridClock initialized: now 1790639270376370 us; error 31 us; skew 500 ppm
Sep 28 23:47:50 dist-test-slave-c3s8 krb5kdc[7555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639270, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:50.379281 7930 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:47:50.380182 7930 webserver.cc:460] Webserver started at http://127.21.112.67:45467/ using document root <none> and password file <none>
I20260928 23:47:50.380797 7930 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:50.380859 7930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:50.381068 7930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:50.383132 7930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "fa1a321b45e444319f005c3d6c24480a"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "63fb23bf282fc8a6cb34ef010b3e47bf"
server_key_iv: "f2a0c93714b072641361b5036de92a0f"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.383687 7930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "fa1a321b45e444319f005c3d6c24480a"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "63fb23bf282fc8a6cb34ef010b3e47bf"
server_key_iv: "f2a0c93714b072641361b5036de92a0f"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.387543 7930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:47:50.390126 7945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.391394 7930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:47:50.391487 7930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "fa1a321b45e444319f005c3d6c24480a"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "63fb23bf282fc8a6cb34ef010b3e47bf"
server_key_iv: "f2a0c93714b072641361b5036de92a0f"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.391621 7930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:50.410897 7930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:50.414160 7930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:50.414390 7930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:50.415011 7930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:50.415901 7930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:50.415946 7930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.416015 7930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:50.416050 7930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.425081 7930 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:34589
I20260928 23:47:50.425220 8058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:34589 every 8 connection(s)
I20260928 23:47:50.426079 7930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:50.435571 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 7930
I20260928 23:47:50.435763 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:47:50.436064 21953 external_mini_cluster.cc:1428] Setting key 49d109950205e28ce11ec52b21146d95
Sep 28 23:47:50 dist-test-slave-c3s8 krb5kdc[7555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639270, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:50.441326 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.428065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:55457 (local address 127.21.112.126:46015)
0928 23:47:50.428221 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.428226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.429158 (+ 932us) server_negotiation.cc:407] Connection header received
0928 23:47:50.430628 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.430630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.430677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.430798 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:50.432074 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.432616 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.434089 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.434421 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.434561 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.437457 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:50.437461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:50.437464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:50.437499 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:50.439218 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:50.439829 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:50.439832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:50.439833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:50.439916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:50.440223 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:50.440225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:50.440227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:50.440461 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:50.440546 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.441033 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.441141 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:47:50.442128 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.428529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46015 (local address 127.21.112.67:55457)
0928 23:47:50.429002 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.429045 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.430430 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.430955 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.430962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.431441 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:50.431938 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.431951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.432788 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.432792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.433934 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.433944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.434605 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.434610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.434801 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.435487 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:50.435505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:50.437253 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:50.439385 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:50.439390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:50.439404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:50.439727 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:50.440002 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:50.440004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:50.440005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:50.440137 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:50.440554 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:50.440559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:50.440796 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.440954 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.441213 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260928 23:47:50.443356 8059 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46015
I20260928 23:47:50.443619 8059 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:50.444175 8059 heartbeater.cc:511] Master 127.21.112.126:46015 requested a full tablet report, sending...
I20260928 23:47:50.445235 7600 ts_manager.cc:194] Registered new tserver with Master: fa1a321b45e444319f005c3d6c24480a (127.21.112.67:34589)
I20260928 23:47:50.445896 7600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:55457
I20260928 23:47:50.450697 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:50.460444 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.452669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.21.112.126:46015)
0928 23:47:50.452804 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.452808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.452996 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:47:50.453057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.453059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.453100 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.453171 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:50.453983 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.454544 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.455425 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.455657 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.455753 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.457304 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:50.457309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:50.457311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:50.457340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:50.458739 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:50.459233 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:50.459235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:50.459236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:50.459316 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:50.459653 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:50.459655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:50.459656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:50.459878 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:50.459993 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.460228 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.460313 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:47:50.463685 7600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:50.466828 7600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:50.482079 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.476025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:34589 (local address 127.0.0.1:53430)
0928 23:47:50.476559 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.476573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.476702 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.478028 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.478031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.478048 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:50.478122 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.478127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.479425 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.479430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.480481 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.480496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.481421 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.481424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.481561 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.481602 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.481749 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.481883 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":90,"threads_started":1}
I20260928 23:47:50.482342 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.476022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.21.112.65:40345)
0928 23:47:50.476403 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.476422 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.476611 (+ 189us) server_negotiation.cc:407] Connection header received
0928 23:47:50.476904 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.476910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.477117 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.477249 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:50.477979 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.479137 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.480739 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.481499 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.481636 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.481791 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.482096 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.482203 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":82,"threads_started":1}
I20260928 23:47:50.482088 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.476546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:34523 (local address 127.0.0.1:42128)
0928 23:47:50.476928 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.476941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.477014 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.477753 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.477756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.477773 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:50.477863 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.477869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.478956 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.478963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.480428 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.480442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.481642 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.481645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.481730 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.481751 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.481807 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.481884 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260928 23:47:50.482587 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.476952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.21.112.66:34523)
0928 23:47:50.477249 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.477262 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.477289 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:47:50.477370 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.477376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.477593 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.477775 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:50.477963 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.478820 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.480564 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.481468 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.482154 (+ 686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.482304 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.482356 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.482453 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260928 23:47:50.482731 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.475841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40345 (local address 127.0.0.1:37884)
0928 23:47:50.476471 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.476521 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.476702 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.477461 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.477467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.477488 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:50.477662 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.477670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.479262 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.479265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.480620 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.480631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.481849 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.481852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.481943 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.481962 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.482531 (+ 569us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.482611 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":232,"thread_start_us":73,"threads_started":1}
I20260928 23:47:50.484941 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.476633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.21.112.67:34589)
0928 23:47:50.477587 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.477598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.477621 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:47:50.477691 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.477695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.477880 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.478057 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:50.478217 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.479299 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.480666 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.481294 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.481421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.481606 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.482628 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.484730 (+ 2102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":152,"threads_started":1}
I20260928 23:47:50.485858 7725 tablet_service.cc:1467] Processing CreateTablet for tablet c4eef5f052204d51bd79a0d2d46e0958 (DEFAULT_TABLE table=test-table [id=24502f11dfa44b63804cd61381ffed17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:50.485864 7859 tablet_service.cc:1467] Processing CreateTablet for tablet c4eef5f052204d51bd79a0d2d46e0958 (DEFAULT_TABLE table=test-table [id=24502f11dfa44b63804cd61381ffed17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:50.487038 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4eef5f052204d51bd79a0d2d46e0958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:50.487040 7859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4eef5f052204d51bd79a0d2d46e0958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:50.488709 7993 tablet_service.cc:1467] Processing CreateTablet for tablet c4eef5f052204d51bd79a0d2d46e0958 (DEFAULT_TABLE table=test-table [id=24502f11dfa44b63804cd61381ffed17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:50.489805 7993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4eef5f052204d51bd79a0d2d46e0958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:50.493525 8076 tablet_bootstrap.cc:492] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Bootstrap starting.
I20260928 23:47:50.495245 8077 tablet_bootstrap.cc:492] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14: Bootstrap starting.
I20260928 23:47:50.496567 8076 tablet_bootstrap.cc:654] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:50.496785 8078 tablet_bootstrap.cc:492] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Bootstrap starting.
I20260928 23:47:50.497406 8077 tablet_bootstrap.cc:654] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:50.498102 8076 log.cc:824] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:50.498175 8077 log.cc:824] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:50.498740 8078 tablet_bootstrap.cc:654] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:50.499782 8078 log.cc:824] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:50.500234 8077 tablet_bootstrap.cc:492] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14: No bootstrap required, opened a new log
I20260928 23:47:50.500240 8076 tablet_bootstrap.cc:492] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: No bootstrap required, opened a new log
I20260928 23:47:50.500442 8077 ts_tablet_manager.cc:1397] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:47:50.500442 8076 ts_tablet_manager.cc:1397] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 23:47:50.501847 8078 tablet_bootstrap.cc:492] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: No bootstrap required, opened a new log
I20260928 23:47:50.502048 8078 ts_tablet_manager.cc:1397] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 23:47:50.504719 8078 raft_consensus.cc:348] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.504632 8076 raft_consensus.cc:348] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.504758 8077 raft_consensus.cc:348] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.504952 8078 raft_consensus.cc:374] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:50.504953 8076 raft_consensus.cc:374] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:50.504971 8077 raft_consensus.cc:374] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:50.505007 8078 raft_consensus.cc:729] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1a321b45e444319f005c3d6c24480a, State: Initialized, Role: FOLLOWER
I20260928 23:47:50.505012 8077 raft_consensus.cc:729] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8db28ba723143ef83261a1df0b20f14, State: Initialized, Role: FOLLOWER
I20260928 23:47:50.505012 8076 raft_consensus.cc:729] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b501a3af4b450ebf6e3b537ac3a989, State: Initialized, Role: FOLLOWER
I20260928 23:47:50.505471 8076 consensus_queue.cc:260] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.505471 8077 consensus_queue.cc:260] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.505482 8078 consensus_queue.cc:260] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.506565 8076 ts_tablet_manager.cc:1428] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 23:47:50.506572 8078 ts_tablet_manager.cc:1428] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 23:47:50.506649 8059 heartbeater.cc:503] Master 127.21.112.126:46015 was elected leader, sending a full tablet report...
I20260928 23:47:50.506661 7925 heartbeater.cc:503] Master 127.21.112.126:46015 was elected leader, sending a full tablet report...
I20260928 23:47:50.506948 7791 heartbeater.cc:503] Master 127.21.112.126:46015 was elected leader, sending a full tablet report...
I20260928 23:47:50.507633 8077 ts_tablet_manager.cc:1428] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:47:50.623168 8082 raft_consensus.cc:482] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:50.623435 8082 raft_consensus.cc:504] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.624675 8082 leader_election.cc:288] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8db28ba723143ef83261a1df0b20f14 (127.21.112.65:40345), 93b501a3af4b450ebf6e3b537ac3a989 (127.21.112.66:34523)
I20260928 23:47:50.629619 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.625083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:46043 (local address 127.21.112.65:40345)
0928 23:47:50.625225 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.625231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.625347 (+ 116us) server_negotiation.cc:407] Connection header received
0928 23:47:50.625465 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.625468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.625526 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.625634 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:50.625979 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.626804 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.628276 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.628882 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.629007 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.629063 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.629384 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.629450 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:47:50.629760 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.625379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:47237 (local address 127.21.112.66:34523)
0928 23:47:50.625512 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.625517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.625755 (+ 238us) server_negotiation.cc:407] Connection header received
0928 23:47:50.625967 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.625971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.626025 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.626155 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:50.626370 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.627047 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.628442 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.629043 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.629155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.629189 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.629573 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.629645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:47:50.629957 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.625264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:34523 (local address 127.21.112.67:47237)
0928 23:47:50.625655 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.625668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.625818 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.626146 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.626148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.626169 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:50.626263 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.626268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.627139 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.627142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.628344 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.628351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.629165 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.629169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.629426 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.629447 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.629524 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.629662 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":58,"thread_start_us":138,"threads_started":1}
I20260928 23:47:50.629679 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.624936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40345 (local address 127.21.112.67:46043)
0928 23:47:50.625149 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.625231 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:50.625360 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:50.625681 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:50.625687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:50.625712 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:50.625854 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.625864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.626917 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.626919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.628148 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:50.628160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.629029 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.629033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:50.629199 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.629263 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 23:47:50.629362 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:47:50.629450 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":29}
I20260928 23:47:50.630707 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4eef5f052204d51bd79a0d2d46e0958" candidate_uuid: "fa1a321b45e444319f005c3d6c24480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b501a3af4b450ebf6e3b537ac3a989" is_pre_election: true
I20260928 23:47:50.630851 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4eef5f052204d51bd79a0d2d46e0958" candidate_uuid: "fa1a321b45e444319f005c3d6c24480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8db28ba723143ef83261a1df0b20f14" is_pre_election: true
I20260928 23:47:50.631071 7879 raft_consensus.cc:2454] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1a321b45e444319f005c3d6c24480a in term 0.
I20260928 23:47:50.631143 7745 raft_consensus.cc:2454] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1a321b45e444319f005c3d6c24480a in term 0.
I20260928 23:47:50.631554 7947 leader_election.cc:302] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b501a3af4b450ebf6e3b537ac3a989, fa1a321b45e444319f005c3d6c24480a; no voters:
I20260928 23:47:50.631829 8082 raft_consensus.cc:2781] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:50.631932 8082 raft_consensus.cc:482] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:50.631990 8082 raft_consensus.cc:3037] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:50.633073 8082 raft_consensus.cc:504] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.633445 8082 leader_election.cc:288] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [CANDIDATE]: Term 1 election: Requested vote from peers c8db28ba723143ef83261a1df0b20f14 (127.21.112.65:40345), 93b501a3af4b450ebf6e3b537ac3a989 (127.21.112.66:34523)
I20260928 23:47:50.633999 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4eef5f052204d51bd79a0d2d46e0958" candidate_uuid: "fa1a321b45e444319f005c3d6c24480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93b501a3af4b450ebf6e3b537ac3a989"
I20260928 23:47:50.634016 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4eef5f052204d51bd79a0d2d46e0958" candidate_uuid: "fa1a321b45e444319f005c3d6c24480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8db28ba723143ef83261a1df0b20f14"
I20260928 23:47:50.634106 7879 raft_consensus.cc:3037] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:50.634192 7745 raft_consensus.cc:3037] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:50.635313 7879 raft_consensus.cc:2454] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1a321b45e444319f005c3d6c24480a in term 1.
I20260928 23:47:50.635550 7745 raft_consensus.cc:2454] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1a321b45e444319f005c3d6c24480a in term 1.
I20260928 23:47:50.635609 7947 leader_election.cc:302] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93b501a3af4b450ebf6e3b537ac3a989, fa1a321b45e444319f005c3d6c24480a; no voters:
I20260928 23:47:50.635828 8082 raft_consensus.cc:2781] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:50.636118 8082 raft_consensus.cc:686] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 1 LEADER]: Becoming Leader. State: Replica: fa1a321b45e444319f005c3d6c24480a, State: Running, Role: LEADER
I20260928 23:47:50.636523 8082 consensus_queue.cc:237] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } }
I20260928 23:47:50.639935 7600 catalog_manager.cc:5495] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a reported cstate change: term changed from 0 to 1, leader changed from <none> to fa1a321b45e444319f005c3d6c24480a (127.21.112.67). New cstate: current_term: 1 leader_uuid: "fa1a321b45e444319f005c3d6c24480a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa1a321b45e444319f005c3d6c24480a" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 34589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 } health_report { overall_health: UNKNOWN } } }
I20260928 23:47:50.676816 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.672846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.21.112.67:34589)
0928 23:47:50.673000 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.673005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.673022 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:47:50.673123 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.673126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.673187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.673300 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:50.673557 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.674361 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.675212 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.675453 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.675539 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.675876 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:50.676283 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:50.676401 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.676580 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.676628 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 23:47:50.677024 8060 tablet.cc:2369] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:50.683002 7879 raft_consensus.cc:1261] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 1 FOLLOWER]: Refusing update from remote peer fa1a321b45e444319f005c3d6c24480a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:50.683002 7745 raft_consensus.cc:1261] T c4eef5f052204d51bd79a0d2d46e0958 P c8db28ba723143ef83261a1df0b20f14 [term 1 FOLLOWER]: Refusing update from remote peer fa1a321b45e444319f005c3d6c24480a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:50.683763 8082 consensus_queue.cc:1035] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93b501a3af4b450ebf6e3b537ac3a989" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 34523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:50.683940 8086 consensus_queue.cc:1035] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8db28ba723143ef83261a1df0b20f14" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:50.692361 8089 mvcc.cc:205] Tried to move back new op lower bound from 7334458452709318656 to 7334458452530032640. Current Snapshot: MvccSnapshot[applied={T|T < 7334458452709318656}]
I20260928 23:47:50.693887 8090 mvcc.cc:205] Tried to move back new op lower bound from 7334458452709318656 to 7334458452530032640. Current Snapshot: MvccSnapshot[applied={T|T < 7334458452709318656}]
I20260928 23:47:50.701529 7600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:50.701805 7600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:50.704372 7600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ece5d02e9f84c12a900c9edf9cb9a8b: Sending DeleteTablet for 3 replicas of tablet c4eef5f052204d51bd79a0d2d46e0958
I20260928 23:47:50.705175 7993 tablet_service.cc:1514] Processing DeleteTablet for tablet c4eef5f052204d51bd79a0d2d46e0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:50 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:53430
I20260928 23:47:50.705229 7725 tablet_service.cc:1514] Processing DeleteTablet for tablet c4eef5f052204d51bd79a0d2d46e0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:50 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:37884
I20260928 23:47:50.705341 7859 tablet_service.cc:1514] Processing DeleteTablet for tablet c4eef5f052204d51bd79a0d2d46e0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:50 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:42128
I20260928 23:47:50.705874 8100 tablet_replica.cc:330] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: stopping tablet replica
I20260928 23:47:50.705924 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7662
I20260928 23:47:50.705740 8098 tablet_replica.cc:330] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: stopping tablet replica
I20260928 23:47:50.707399 8098 raft_consensus.cc:2229] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:47:50.707553 8100 raft_consensus.cc:2229] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:50.707886 8100 raft_consensus.cc:2258] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:50.707886 8098 raft_consensus.cc:2258] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:50.709407 8098 ts_tablet_manager.cc:1905] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:50.709407 8100 ts_tablet_manager.cc:1905] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:50.712124 8098 ts_tablet_manager.cc:1918] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:50.712124 8100 ts_tablet_manager.cc:1918] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:50.712224 8100 log.cc:1196] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/c4eef5f052204d51bd79a0d2d46e0958
I20260928 23:47:50.712339 8098 log.cc:1196] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/c4eef5f052204d51bd79a0d2d46e0958
I20260928 23:47:50.712558 8100 ts_tablet_manager.cc:1939] T c4eef5f052204d51bd79a0d2d46e0958 P 93b501a3af4b450ebf6e3b537ac3a989: Deleting consensus metadata
I20260928 23:47:50.712783 8098 ts_tablet_manager.cc:1939] T c4eef5f052204d51bd79a0d2d46e0958 P fa1a321b45e444319f005c3d6c24480a: Deleting consensus metadata
W20260928 23:47:50.713258 7585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:40345 (error 108)
I20260928 23:47:50.714069 7587 catalog_manager.cc:4841] TS fa1a321b45e444319f005c3d6c24480a (127.21.112.67:34589): tablet c4eef5f052204d51bd79a0d2d46e0958 (table test-table [id=24502f11dfa44b63804cd61381ffed17]) successfully deleted
I20260928 23:47:50.714145 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7796
I20260928 23:47:50.714069 7585 catalog_manager.cc:4841] TS 93b501a3af4b450ebf6e3b537ac3a989 (127.21.112.66:34523): tablet c4eef5f052204d51bd79a0d2d46e0958 (table test-table [id=24502f11dfa44b63804cd61381ffed17]) successfully deleted
I20260928 23:47:50.715106 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.714794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40345 (local address 127.0.0.1:37894)
0928 23:47:50.714973 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:50.715032 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:40345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260928 23:47:50.715315 7585 catalog_manager.cc:4584] TS c8db28ba723143ef83261a1df0b20f14 (127.21.112.65:40345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4eef5f052204d51bd79a0d2d46e0958: Network error: Client connection negotiation failed: client connection to 127.21.112.65:40345: connect: Connection refused (error 111)
I20260928 23:47:50.722081 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7930
I20260928 23:47:50.727312 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 7568
2026-09-28T23:47:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2431 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T23:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:50Z Disabled control of system clock
I20260928 23:47:50.759752 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46143
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:46143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:47:50.877327 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:50.877635 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:50.877672 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:50.881672 8105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:50.881729 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:50.881744 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:50.881762 8105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:50.881776 8105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:50.886612 8105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:46143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:46143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:50.887676 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:50.889055 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:47:50.897869 8105 server_base.cc:1034] running on GCE node
W20260928 23:47:50.897758 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:50.897804 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:50.897773 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:50.898547 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:50.899678 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:50.900840 8105 hybrid_clock.cc:648] HybridClock initialized: now 1790639270900807 us; error 46 us; skew 500 ppm
I20260928 23:47:50.902791 8105 webserver.cc:460] Webserver started at http://127.21.112.126:38755/ using document root <none> and password file <none>
I20260928 23:47:50.903327 8105 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:50.903384 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:50.903545 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:50.905614 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "d93e9b6f1ab8421c90cbe7c661c2c453"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "d66dcd3a6b2ba27c3427e7ba494a5ea6"
server_key_iv: "0186e6bd6c1dd75d33e66475f60d6239"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.906136 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "d93e9b6f1ab8421c90cbe7c661c2c453"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "d66dcd3a6b2ba27c3427e7ba494a5ea6"
server_key_iv: "0186e6bd6c1dd75d33e66475f60d6239"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.909819 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 23:47:50.912326 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:50.913751 8105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:50.913877 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "d93e9b6f1ab8421c90cbe7c661c2c453"
format_stamp: "Formatted at 2026-09-28 23:47:50 on dist-test-slave-c3s8"
server_key: "d66dcd3a6b2ba27c3427e7ba494a5ea6"
server_key_iv: "0186e6bd6c1dd75d33e66475f60d6239"
server_key_version: "encryptionkey@0"
I20260928 23:47:50.913986 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:50.923874 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:50.924505 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:50.924707 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:50.931568 8105 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:46143
I20260928 23:47:50.931599 8171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:46143 every 8 connection(s)
I20260928 23:47:50.932480 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:50.935608 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:50.936296 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8105
I20260928 23:47:50.936444 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:50.936728 21953 external_mini_cluster.cc:1428] Setting key fc47e710410188561e0dcd906360748c
I20260928 23:47:50.941684 8172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453: Bootstrap starting.
I20260928 23:47:50.944396 8172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:50.945210 8172 log.cc:824] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:50.946766 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:50.938113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.21.112.126:46143)
0928 23:47:50.938778 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:50.938795 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:50.938831 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:47:50.939586 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:50.939605 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:50.940015 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:50.940320 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:50.940671 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.941768 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.942986 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:50.943557 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:50.943686 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:50.944707 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:50.944713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:50.944726 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:50.945143 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:50.945205 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:50.945235 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:50.945375 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:50.945522 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 23:47:50.945883 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":137,"threads_started":1}
I20260928 23:47:50.947324 8172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453: No bootstrap required, opened a new log
I20260928 23:47:50.951261 8172 raft_consensus.cc:348] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46143 } }
I20260928 23:47:50.951550 8172 raft_consensus.cc:374] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:50.951606 8172 raft_consensus.cc:729] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93e9b6f1ab8421c90cbe7c661c2c453, State: Initialized, Role: FOLLOWER
I20260928 23:47:50.952076 8172 consensus_queue.cc:260] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46143 } }
I20260928 23:47:50.952247 8172 raft_consensus.cc:388] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:50.952328 8172 raft_consensus.cc:482] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:50.952423 8172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:50.953289 8172 raft_consensus.cc:504] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46143 } }
I20260928 23:47:50.953678 8172 leader_election.cc:302] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d93e9b6f1ab8421c90cbe7c661c2c453; no voters:
I20260928 23:47:50.953944 8172 leader_election.cc:288] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:50.954099 8177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:50.954411 8177 raft_consensus.cc:686] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [term 1 LEADER]: Becoming Leader. State: Replica: d93e9b6f1ab8421c90cbe7c661c2c453, State: Running, Role: LEADER
I20260928 23:47:50.954730 8177 consensus_queue.cc:237] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46143 } }
I20260928 23:47:50.955201 8172 sys_catalog.cc:564] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:50.956327 8179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93e9b6f1ab8421c90cbe7c661c2c453. Latest consensus state: current_term: 1 leader_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46143 } } }
I20260928 23:47:50.956472 8179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:50.956848 8186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:50.957288 8178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d93e9b6f1ab8421c90cbe7c661c2c453" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 46143 } } }
I20260928 23:47:50.957404 8178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:50.960016 8186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:50.966015 8186 catalog_manager.cc:1347] Generated new cluster ID: b9abc85cfe7c449b823cc78cfdaa3ee8
I20260928 23:47:50.966095 8186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:50.992133 8186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:50.993188 8186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:51.011034 8186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453: Generated new TSK 0
I20260928 23:47:51.011978 8186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:47:51.023370 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46143
--builtin_ntp_servers=127.21.112.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:47:51.137961 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:51.138244 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:51.138283 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:51.142472 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:51.142529 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:51.142616 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:51.147403 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:51.148515 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:51.150063 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:51.160475 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:51.160467 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:51.160533 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:51.160593 8196 server_base.cc:1034] running on GCE node
I20260928 23:47:51.161134 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:51.161917 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:51.163130 8196 hybrid_clock.cc:648] HybridClock initialized: now 1790639271163090 us; error 54 us; skew 500 ppm
I20260928 23:47:51.165086 8196 webserver.cc:460] Webserver started at http://127.21.112.65:38381/ using document root <none> and password file <none>
I20260928 23:47:51.165827 8196 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:51.165925 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:51.166163 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:51.168275 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "433383ee53114c90941ec2af3436ae63"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "0850963812c96f25e835acdbc763e00f"
server_key_iv: "e6fa42261b165c6cfc9729957b61699a"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.168864 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "433383ee53114c90941ec2af3436ae63"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "0850963812c96f25e835acdbc763e00f"
server_key_iv: "e6fa42261b165c6cfc9729957b61699a"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.172724 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:47:51.175688 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.177034 8196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:51.177186 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "433383ee53114c90941ec2af3436ae63"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "0850963812c96f25e835acdbc763e00f"
server_key_iv: "e6fa42261b165c6cfc9729957b61699a"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.177318 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:51.190702 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:51.191416 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:51.191615 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:51.192221 8196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:51.193178 8196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:51.193251 8196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.193359 8196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:51.193403 8196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.203421 8196 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:41311
I20260928 23:47:51.203442 8323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:41311 every 8 connection(s)
I20260928 23:47:51.204398 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:51.210101 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8196
I20260928 23:47:51.210222 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:47:51.210518 21953 external_mini_cluster.cc:1428] Setting key 227abc1238e3450fc21f86f1ed49ca25
I20260928 23:47:51.212846 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46143
--builtin_ntp_servers=127.21.112.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:47:51.215307 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.206640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:36069 (local address 127.21.112.126:46143)
0928 23:47:51.206824 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.206830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.208012 (+ 1182us) server_negotiation.cc:407] Connection header received
0928 23:47:51.209472 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.209475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.209526 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.209634 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:51.210754 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.211414 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.212834 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.213218 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.213364 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.214679 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:51.214687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:51.214692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:51.214712 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:51.214773 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:51.214788 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:51.214886 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.215031 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.215111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 23:47:51.216166 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.207105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46143 (local address 127.21.112.65:36069)
0928 23:47:51.207837 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.207881 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.209235 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.209846 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.209854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.209963 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:51.210574 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.210597 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.211547 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.211550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.212699 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.212710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.213414 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.213417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.213669 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.214392 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:51.214394 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:51.214437 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:51.214504 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:51.214884 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:51.214890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:51.214894 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.214993 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.215231 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260928 23:47:51.217582 8324 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46143
I20260928 23:47:51.217818 8324 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:51.218444 8324 heartbeater.cc:511] Master 127.21.112.126:46143 requested a full tablet report, sending...
I20260928 23:47:51.220037 8136 ts_manager.cc:194] Registered new tserver with Master: 433383ee53114c90941ec2af3436ae63 (127.21.112.65:41311)
I20260928 23:47:51.221159 8136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.65:36069
W20260928 23:47:51.330669 8327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:51.330965 8327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:51.331005 8327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:51.335207 8327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:51.335269 8327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:51.335362 8327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:51.340418 8327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:51.341602 8327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:51.343031 8327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:51.353021 8333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:51.353060 8332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:51.353215 8327 server_base.cc:1034] running on GCE node
W20260928 23:47:51.353024 8335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:51.353883 8327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:51.354663 8327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:51.355844 8327 hybrid_clock.cc:648] HybridClock initialized: now 1790639271355738 us; error 105 us; skew 500 ppm
I20260928 23:47:51.358129 8327 webserver.cc:460] Webserver started at http://127.21.112.66:40929/ using document root <none> and password file <none>
I20260928 23:47:51.358747 8327 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:51.358815 8327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:51.359001 8327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:51.361151 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "bdf40b9999d6459dae32ffb959c74baa"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "15fd3a4cbafe0a2753cf24695284ac81"
server_key_iv: "f13c0ef2f64da5f803c9d52a51ffa804"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.361747 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "bdf40b9999d6459dae32ffb959c74baa"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "15fd3a4cbafe0a2753cf24695284ac81"
server_key_iv: "f13c0ef2f64da5f803c9d52a51ffa804"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.365635 8327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:47:51.368201 8341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.369648 8327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:51.369787 8327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "bdf40b9999d6459dae32ffb959c74baa"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "15fd3a4cbafe0a2753cf24695284ac81"
server_key_iv: "f13c0ef2f64da5f803c9d52a51ffa804"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.369918 8327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:51.382714 8327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:51.383430 8327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:51.383633 8327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:51.384230 8327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:51.385183 8327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:51.385257 8327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.385325 8327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:51.385375 8327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.394971 8327 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:41477
I20260928 23:47:51.395015 8454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:41477 every 8 connection(s)
I20260928 23:47:51.395967 8327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:51.400465 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8327
I20260928 23:47:51.400629 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:47:51.400949 21953 external_mini_cluster.cc:1428] Setting key 3fd7106690d4200d79e50e4378ae86ab
I20260928 23:47:51.403247 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46143
--builtin_ntp_servers=127.21.112.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:47:51.406467 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.397957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:51111 (local address 127.21.112.126:46143)
0928 23:47:51.398095 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.398099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.399190 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 23:47:51.400578 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.400581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.400643 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.400752 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:51.401893 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.402490 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.404231 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.404493 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.404579 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.405872 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:51.405879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:51.405885 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:51.405902 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:51.405930 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:51.405943 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:51.406024 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.406172 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.406245 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:47:51.407325 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.398383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46143 (local address 127.21.112.66:51111)
0928 23:47:51.399056 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.399096 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.400352 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.400959 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.400966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.401086 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:51.401734 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.401752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.402624 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.402628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.404097 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.404112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.404664 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.404681 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.404895 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.405597 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:51.405599 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:51.405642 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:51.405716 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:51.406030 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:51.406037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:51.406041 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.406140 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.406382 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260928 23:47:51.408640 8455 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46143
I20260928 23:47:51.408914 8455 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:51.409500 8455 heartbeater.cc:511] Master 127.21.112.126:46143 requested a full tablet report, sending...
I20260928 23:47:51.410735 8136 ts_manager.cc:194] Registered new tserver with Master: bdf40b9999d6459dae32ffb959c74baa (127.21.112.66:41477)
I20260928 23:47:51.411412 8136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.66:51111
W20260928 23:47:51.518139 8458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:51.518522 8458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:51.518611 8458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:51.523192 8458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:51.523300 8458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:51.523401 8458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:51.528366 8458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:46143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:51.529752 8458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:51.531267 8458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:51.540997 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:51.540983 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:51.541141 8458 server_base.cc:1034] running on GCE node
W20260928 23:47:51.540987 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:51.541706 8458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:51.542356 8458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:51.543572 8458 hybrid_clock.cc:648] HybridClock initialized: now 1790639271543520 us; error 67 us; skew 500 ppm
I20260928 23:47:51.545696 8458 webserver.cc:460] Webserver started at http://127.21.112.67:44971/ using document root <none> and password file <none>
I20260928 23:47:51.546293 8458 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:51.546388 8458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:51.546605 8458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:51.548774 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "5d2fe5086249868a45f6f8d9fecbd866"
server_key_iv: "0b7c11fdf9f0b402a0c710768f6d4946"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.549432 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "5d2fe5086249868a45f6f8d9fecbd866"
server_key_iv: "0b7c11fdf9f0b402a0c710768f6d4946"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.553481 8458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 23:47:51.556341 8472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.557755 8458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:51.557890 8458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea"
format_stamp: "Formatted at 2026-09-28 23:47:51 on dist-test-slave-c3s8"
server_key: "5d2fe5086249868a45f6f8d9fecbd866"
server_key_iv: "0b7c11fdf9f0b402a0c710768f6d4946"
server_key_version: "encryptionkey@0"
I20260928 23:47:51.558009 8458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:51.576081 8458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:51.576817 8458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:51.577019 8458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:51.577687 8458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:51.578652 8458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:51.578722 8458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.578794 8458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:51.578830 8458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:51.588383 8458 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:45555
I20260928 23:47:51.588420 8585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:45555 every 8 connection(s)
I20260928 23:47:51.589340 8458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:51.590333 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8458
I20260928 23:47:51.590435 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:47:51.590696 21953 external_mini_cluster.cc:1428] Setting key 7705cf224863aca06fdcd2f3d4e1f24c
I20260928 23:47:51.600472 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.592322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:51029 (local address 127.21.112.126:46143)
0928 23:47:51.592484 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.592489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.593822 (+ 1333us) server_negotiation.cc:407] Connection header received
0928 23:47:51.595371 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.595374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.595425 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.595498 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:51.596525 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.597054 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.598377 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.598603 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.598689 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.599881 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:51.599884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:51.599886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:51.599900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:51.599920 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:51.599927 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:51.599999 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.600226 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.600347 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:47:51.601320 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.592807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:46143 (local address 127.21.112.67:51029)
0928 23:47:51.593647 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.593705 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.595148 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.595702 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.595709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.595819 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:51.596356 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.596374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.597189 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.597192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.598249 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.598260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.598755 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.598758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.598919 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.599561 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:51.599563 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:51.599608 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:51.599704 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:51.600044 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:51.600053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:51.600065 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.600186 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.600427 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":154,"threads_started":1}
I20260928 23:47:51.602541 8586 heartbeater.cc:347] Connected to a master server at 127.21.112.126:46143
I20260928 23:47:51.602766 8586 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:51.603261 8586 heartbeater.cc:511] Master 127.21.112.126:46143 requested a full tablet report, sending...
I20260928 23:47:51.604403 8135 ts_manager.cc:194] Registered new tserver with Master: c6f04c6e5e8e4314b0c6eb96b101b6ea (127.21.112.67:45555)
I20260928 23:47:51.605027 8135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.67:51029
I20260928 23:47:51.605098 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:51.611356 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.607274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.21.112.126:46143)
0928 23:47:51.607399 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.607403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.607599 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:47:51.607643 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.607646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.607683 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.607763 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:51.608104 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.608606 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.609619 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.609855 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.609934 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.610883 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:51.610886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:51.610888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:51.610900 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:51.610920 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:51.610928 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:51.611035 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.611190 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.611233 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:47:51.614672 8135 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:51.617741 8135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:51.632951 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.626685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.21.112.65:41311)
0928 23:47:51.627058 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.627070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.627502 (+ 432us) server_negotiation.cc:407] Connection header received
0928 23:47:51.627582 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.627588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.627807 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.628336 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:51.628471 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.629661 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.631471 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.632187 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.632318 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.632506 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.632668 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.632778 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260928 23:47:51.632954 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.627285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45555 (local address 127.0.0.1:58476)
0928 23:47:51.628158 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.628171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.628251 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.628724 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.628730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.628751 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:51.628825 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.628830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.630127 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.630130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.631234 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.631250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.632334 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.632338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.632446 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.632510 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.632627 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.632787 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":53,"thread_start_us":84,"threads_started":1}
I20260928 23:47:51.632956 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.626556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:41311 (local address 127.0.0.1:44068)
0928 23:47:51.627202 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.627253 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.627387 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.627981 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.627986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.628006 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:51.628167 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.628176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.629789 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.629792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.631343 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.631355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.632395 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.632399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.632471 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.632522 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.632627 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.632787 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":202,"thread_start_us":92,"threads_started":1}
I20260928 23:47:51.633421 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.627448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.21.112.67:45555)
0928 23:47:51.628068 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.628079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.628345 (+ 266us) server_negotiation.cc:407] Connection header received
0928 23:47:51.628424 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.628429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.628607 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.628761 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:51.628918 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.630011 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.631362 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.632188 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.632313 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.632506 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.633157 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.633262 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":139,"threads_started":1}
I20260928 23:47:51.634212 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.626568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:41477 (local address 127.0.0.1:40516)
0928 23:47:51.627903 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.627918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.627998 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.628542 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.628546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.628563 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:51.628673 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.628683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.629777 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.629780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.631350 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.631355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.633775 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.633778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.633890 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.633910 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.633999 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.634068 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":110,"thread_start_us":110,"threads_started":1}
I20260928 23:47:51.634311 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.627348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.21.112.66:41477)
0928 23:47:51.627883 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.627895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.628124 (+ 229us) server_negotiation.cc:407] Connection header received
0928 23:47:51.628207 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.628214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.628406 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.628567 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:51.628806 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.629660 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.632574 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.633380 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.633489 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.633707 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.634035 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.634129 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":126,"threads_started":1}
I20260928 23:47:51.636670 8258 tablet_service.cc:1467] Processing CreateTablet for tablet ae3f1b662bd6401ab456b574ac9c1cc7 (DEFAULT_TABLE table=test-table [id=ae28bc7c6dcc44b3b2308273360c4aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:51.636806 8389 tablet_service.cc:1467] Processing CreateTablet for tablet ae3f1b662bd6401ab456b574ac9c1cc7 (DEFAULT_TABLE table=test-table [id=ae28bc7c6dcc44b3b2308273360c4aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:51.636799 8520 tablet_service.cc:1467] Processing CreateTablet for tablet ae3f1b662bd6401ab456b574ac9c1cc7 (DEFAULT_TABLE table=test-table [id=ae28bc7c6dcc44b3b2308273360c4aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:51.637775 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae3f1b662bd6401ab456b574ac9c1cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:51.637835 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae3f1b662bd6401ab456b574ac9c1cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:51.637835 8520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae3f1b662bd6401ab456b574ac9c1cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:51.644256 8603 tablet_bootstrap.cc:492] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: Bootstrap starting.
I20260928 23:47:51.645848 8604 tablet_bootstrap.cc:492] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Bootstrap starting.
I20260928 23:47:51.646538 8603 tablet_bootstrap.cc:654] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:51.647264 8603 log.cc:824] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:51.647692 8604 tablet_bootstrap.cc:654] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:51.648003 8605 tablet_bootstrap.cc:492] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Bootstrap starting.
I20260928 23:47:51.648505 8604 log.cc:824] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:51.649469 8603 tablet_bootstrap.cc:492] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: No bootstrap required, opened a new log
I20260928 23:47:51.649706 8603 ts_tablet_manager.cc:1397] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 23:47:51.650027 8605 tablet_bootstrap.cc:654] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:51.650749 8605 log.cc:824] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:51.652583 8604 tablet_bootstrap.cc:492] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: No bootstrap required, opened a new log
I20260928 23:47:51.652758 8605 tablet_bootstrap.cc:492] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: No bootstrap required, opened a new log
I20260928 23:47:51.652798 8604 ts_tablet_manager.cc:1397] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 23:47:51.652944 8605 ts_tablet_manager.cc:1397] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:47:51.654002 8603 raft_consensus.cc:348] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.654304 8603 raft_consensus.cc:374] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:51.654372 8603 raft_consensus.cc:729] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433383ee53114c90941ec2af3436ae63, State: Initialized, Role: FOLLOWER
I20260928 23:47:51.654848 8603 consensus_queue.cc:260] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.656009 8603 ts_tablet_manager.cc:1428] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 23:47:51.656380 8324 heartbeater.cc:503] Master 127.21.112.126:46143 was elected leader, sending a full tablet report...
I20260928 23:47:51.657397 8605 raft_consensus.cc:348] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.657718 8605 raft_consensus.cc:374] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:51.657776 8605 raft_consensus.cc:729] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f04c6e5e8e4314b0c6eb96b101b6ea, State: Initialized, Role: FOLLOWER
I20260928 23:47:51.657966 8604 raft_consensus.cc:348] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.658217 8604 raft_consensus.cc:374] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:51.658277 8604 raft_consensus.cc:729] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf40b9999d6459dae32ffb959c74baa, State: Initialized, Role: FOLLOWER
I20260928 23:47:51.658270 8605 consensus_queue.cc:260] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.658712 8604 consensus_queue.cc:260] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.659351 8605 ts_tablet_manager.cc:1428] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:47:51.659368 8455 heartbeater.cc:503] Master 127.21.112.126:46143 was elected leader, sending a full tablet report...
I20260928 23:47:51.659556 8586 heartbeater.cc:503] Master 127.21.112.126:46143 was elected leader, sending a full tablet report...
I20260928 23:47:51.660708 8604 ts_tablet_manager.cc:1428] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 23:47:51.673106 8609 raft_consensus.cc:482] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:51.673416 8609 raft_consensus.cc:504] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.674825 8609 leader_election.cc:288] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdf40b9999d6459dae32ffb959c74baa (127.21.112.66:41477), c6f04c6e5e8e4314b0c6eb96b101b6ea (127.21.112.67:45555)
I20260928 23:47:51.679956 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.675544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:60391 (local address 127.21.112.67:45555)
0928 23:47:51.675681 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.675685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.675697 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:47:51.675747 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.675749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.675786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.675878 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:51.676124 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.676831 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.678450 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.679109 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.679241 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.679293 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.679744 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.679830 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:47:51.679956 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.675133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:33117 (local address 127.21.112.66:41477)
0928 23:47:51.675256 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.675260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.675274 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:47:51.675399 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.675402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.675452 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.675520 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:51.675893 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.676681 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.678450 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.679107 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.679303 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.679351 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.679748 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.679830 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:47:51.679955 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.674986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:41477 (local address 127.21.112.65:33117)
0928 23:47:51.675158 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.675174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.675266 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.675641 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.675645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.675662 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:51.675774 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.675780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.676823 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.676826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.678295 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.678308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.679317 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.679321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.679517 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.679594 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.679676 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.679790 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":168}
I20260928 23:47:51.679962 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.675054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:45555 (local address 127.21.112.65:60391)
0928 23:47:51.675502 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.675516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:51.675594 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:51.675899 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:51.675902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:51.675919 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:51.676021 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.676025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.676939 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.676942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.678295 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:51.678308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.679317 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.679321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:51.679521 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.679590 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 23:47:51.679676 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:47:51.679790 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":143,"thread_start_us":119,"threads_started":1}
I20260928 23:47:51.680651 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3f1b662bd6401ab456b574ac9c1cc7" candidate_uuid: "433383ee53114c90941ec2af3436ae63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf40b9999d6459dae32ffb959c74baa" is_pre_election: true
I20260928 23:47:51.680822 8540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3f1b662bd6401ab456b574ac9c1cc7" candidate_uuid: "433383ee53114c90941ec2af3436ae63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" is_pre_election: true
I20260928 23:47:51.680914 8409 raft_consensus.cc:2454] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433383ee53114c90941ec2af3436ae63 in term 0.
I20260928 23:47:51.681128 8540 raft_consensus.cc:2454] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433383ee53114c90941ec2af3436ae63 in term 0.
I20260928 23:47:51.681381 8214 leader_election.cc:302] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433383ee53114c90941ec2af3436ae63, bdf40b9999d6459dae32ffb959c74baa; no voters:
I20260928 23:47:51.681713 8609 raft_consensus.cc:2781] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:51.681818 8609 raft_consensus.cc:482] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:51.681874 8609 raft_consensus.cc:3037] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:51.683045 8609 raft_consensus.cc:504] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.683403 8609 leader_election.cc:288] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [CANDIDATE]: Term 1 election: Requested vote from peers bdf40b9999d6459dae32ffb959c74baa (127.21.112.66:41477), c6f04c6e5e8e4314b0c6eb96b101b6ea (127.21.112.67:45555)
I20260928 23:47:51.683812 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3f1b662bd6401ab456b574ac9c1cc7" candidate_uuid: "433383ee53114c90941ec2af3436ae63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdf40b9999d6459dae32ffb959c74baa"
I20260928 23:47:51.683854 8540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3f1b662bd6401ab456b574ac9c1cc7" candidate_uuid: "433383ee53114c90941ec2af3436ae63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea"
I20260928 23:47:51.683961 8540 raft_consensus.cc:3037] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:51.683960 8409 raft_consensus.cc:3037] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:51.685310 8409 raft_consensus.cc:2454] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433383ee53114c90941ec2af3436ae63 in term 1.
I20260928 23:47:51.685310 8540 raft_consensus.cc:2454] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433383ee53114c90941ec2af3436ae63 in term 1.
I20260928 23:47:51.685657 8214 leader_election.cc:302] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433383ee53114c90941ec2af3436ae63, bdf40b9999d6459dae32ffb959c74baa; no voters:
I20260928 23:47:51.685896 8609 raft_consensus.cc:2781] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:51.686103 8609 raft_consensus.cc:686] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [term 1 LEADER]: Becoming Leader. State: Replica: 433383ee53114c90941ec2af3436ae63, State: Running, Role: LEADER
I20260928 23:47:51.686527 8609 consensus_queue.cc:237] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } }
I20260928 23:47:51.689790 8135 catalog_manager.cc:5495] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 433383ee53114c90941ec2af3436ae63 (127.21.112.65). New cstate: current_term: 1 leader_uuid: "433383ee53114c90941ec2af3436ae63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433383ee53114c90941ec2af3436ae63" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 41311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 } health_report { overall_health: UNKNOWN } } }
W20260928 23:47:51.705803 8325 tablet.cc:2369] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:51.712978 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.709065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.21.112.65:41311)
0928 23:47:51.709317 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:51.709323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:51.709395 (+ 72us) server_negotiation.cc:407] Connection header received
0928 23:47:51.709612 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:51.709615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:51.709673 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:51.709800 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:51.710057 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.710560 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.711451 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:51.711680 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:51.711802 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:51.712106 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:51.712517 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:51.712679 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:51.712774 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:47:51.712812 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 23:47:51.719441 8540 raft_consensus.cc:1261] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 1 FOLLOWER]: Refusing update from remote peer 433383ee53114c90941ec2af3436ae63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:51.719441 8409 raft_consensus.cc:1261] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 1 FOLLOWER]: Refusing update from remote peer 433383ee53114c90941ec2af3436ae63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:51.720235 8609 consensus_queue.cc:1035] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f04c6e5e8e4314b0c6eb96b101b6ea" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:51.720456 8613 consensus_queue.cc:1035] T ae3f1b662bd6401ab456b574ac9c1cc7 P 433383ee53114c90941ec2af3436ae63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdf40b9999d6459dae32ffb959c74baa" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:51.728953 8618 mvcc.cc:205] Tried to move back new op lower bound from 7334458456953823232 to 7334458456831102976. Current Snapshot: MvccSnapshot[applied={T|T < 7334458456953823232}]
I20260928 23:47:51.729987 8617 mvcc.cc:205] Tried to move back new op lower bound from 7334458456953823232 to 7334458456831102976. Current Snapshot: MvccSnapshot[applied={T|T < 7334458456953823232 or (T in {7334458456953823232})}]
I20260928 23:47:51.737764 8134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:51.737950 8134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:51.740428 8134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d93e9b6f1ab8421c90cbe7c661c2c453: Sending DeleteTablet for 3 replicas of tablet ae3f1b662bd6401ab456b574ac9c1cc7
I20260928 23:47:51.741273 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet ae3f1b662bd6401ab456b574ac9c1cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:51 UTC) from {username='slave'} at 127.0.0.1:40516
I20260928 23:47:51.741254 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet ae3f1b662bd6401ab456b574ac9c1cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:51 UTC) from {username='slave'} at 127.0.0.1:44068
I20260928 23:47:51.741411 8520 tablet_service.cc:1514] Processing DeleteTablet for tablet ae3f1b662bd6401ab456b574ac9c1cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:51 UTC) from {username='slave'} at 127.0.0.1:58476
I20260928 23:47:51.741839 8626 tablet_replica.cc:330] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: stopping tablet replica
I20260928 23:47:51.741942 8628 tablet_replica.cc:330] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: stopping tablet replica
I20260928 23:47:51.742183 8626 raft_consensus.cc:2229] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:51.742182 8628 raft_consensus.cc:2229] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:51.742221 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8196
I20260928 23:47:51.742499 8626 raft_consensus.cc:2258] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:51.743861 8628 raft_consensus.cc:2258] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:51.744428 8626 ts_tablet_manager.cc:1905] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:51.745368 8628 ts_tablet_manager.cc:1905] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:51.747279 8626 ts_tablet_manager.cc:1918] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:51.747377 8626 log.cc:1196] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/ae3f1b662bd6401ab456b574ac9c1cc7
I20260928 23:47:51.747673 8626 ts_tablet_manager.cc:1939] T ae3f1b662bd6401ab456b574ac9c1cc7 P bdf40b9999d6459dae32ffb959c74baa: Deleting consensus metadata
I20260928 23:47:51.748138 8628 ts_tablet_manager.cc:1918] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:51.748229 8628 log.cc:1196] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/ae3f1b662bd6401ab456b574ac9c1cc7
I20260928 23:47:51.748507 8628 ts_tablet_manager.cc:1939] T ae3f1b662bd6401ab456b574ac9c1cc7 P c6f04c6e5e8e4314b0c6eb96b101b6ea: Deleting consensus metadata
W20260928 23:47:51.748689 8120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:41311 (error 108)
I20260928 23:47:51.749037 8123 catalog_manager.cc:4841] TS bdf40b9999d6459dae32ffb959c74baa (127.21.112.66:41477): tablet ae3f1b662bd6401ab456b574ac9c1cc7 (table test-table [id=ae28bc7c6dcc44b3b2308273360c4aad]) successfully deleted
I20260928 23:47:51.749565 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8327
I20260928 23:47:51.749809 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:51.749484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:41311 (local address 127.0.0.1:44092)
0928 23:47:51.749641 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:51.749715 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:41311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 23:47:51.750396 8120 catalog_manager.cc:4584] TS 433383ee53114c90941ec2af3436ae63 (127.21.112.65:41311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae3f1b662bd6401ab456b574ac9c1cc7: Network error: Client connection negotiation failed: client connection to 127.21.112.65:41311: connect: Connection refused (error 111)
I20260928 23:47:51.751369 8120 catalog_manager.cc:4841] TS c6f04c6e5e8e4314b0c6eb96b101b6ea (127.21.112.67:45555): tablet ae3f1b662bd6401ab456b574ac9c1cc7 (table test-table [id=ae28bc7c6dcc44b3b2308273360c4aad]) successfully deleted
I20260928 23:47:51.755317 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8458
I20260928 23:47:51.761111 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8105
2026-09-28T23:47:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:51 dist-test-slave-c3s8 krb5kdc[8632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:51 dist-test-slave-c3s8 krb5kdc[8632](info): set up 2 sockets
Sep 28 23:47:51 dist-test-slave-c3s8 krb5kdc[8632](info): commencing operation
krb5kdc: starting...
W20260928 23:47:52.806104 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:47:52 dist-test-slave-c3s8 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:47:52.933216 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:45923
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:45923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:53.046376 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:53.046650 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:53.046691 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:53.050837 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:53.050900 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:53.050920 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:53.050936 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:53.050952 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:53.056411 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:45923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:45923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:53.057466 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:53.058568 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:53.067442 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:53.067543 8645 server_base.cc:1034] running on GCE node
W20260928 23:47:53.067456 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:53.067459 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:53.068370 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:53.069691 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:53.070910 8645 hybrid_clock.cc:648] HybridClock initialized: now 1790639273070896 us; error 25 us; skew 500 ppm
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:53.074669 8645 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:47:53.075987 8645 webserver.cc:460] Webserver started at http://127.21.112.126:34349/ using document root <none> and password file <none>
I20260928 23:47:53.076761 8645 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:53.076851 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:53.077118 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:53.080087 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "a87e11991ecb430fb1daecce6f2a14f4"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "8b77f2249415918370f9669d91704677"
server_key_iv: "9f6282f3229d2b8a7e5342c38b258a26"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.080932 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "a87e11991ecb430fb1daecce6f2a14f4"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "8b77f2249415918370f9669d91704677"
server_key_iv: "9f6282f3229d2b8a7e5342c38b258a26"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.087452 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260928 23:47:53.091043 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.092602 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:47:53.092784 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "a87e11991ecb430fb1daecce6f2a14f4"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "8b77f2249415918370f9669d91704677"
server_key_iv: "9f6282f3229d2b8a7e5342c38b258a26"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.092914 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:53.104995 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:53.109781 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:53.110018 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:53.117573 8645 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:45923
I20260928 23:47:53.117630 8712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:45923 every 8 connection(s)
I20260928 23:47:53.118724 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:53.119900 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8645
I20260928 23:47:53.120067 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:53.120347 21953 external_mini_cluster.cc:1428] Setting key a15dd80ebe3fbba95ad34cb7bb5a6c5d
I20260928 23:47:53.122146 8713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:53.129278 8713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4: Bootstrap starting.
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:53.132043 8713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:53.132980 8713 log.cc:824] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:53.136790 8713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4: No bootstrap required, opened a new log
I20260928 23:47:53.137725 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.121787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.21.112.126:45923)
0928 23:47:53.122228 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.122248 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.122289 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:47:53.123175 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.123209 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.123649 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.124029 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:53.124915 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.126513 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.127501 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.128264 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.128443 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.131222 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:53.131228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:53.131245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:53.131285 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:53.133583 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.134244 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.134253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.134263 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.134426 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.134809 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.134814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.134817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.135443 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:53.136242 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.136398 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.136698 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260928 23:47:53.141822 8713 raft_consensus.cc:348] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87e11991ecb430fb1daecce6f2a14f4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45923 } }
I20260928 23:47:53.142174 8713 raft_consensus.cc:374] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:53.142233 8713 raft_consensus.cc:729] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87e11991ecb430fb1daecce6f2a14f4, State: Initialized, Role: FOLLOWER
I20260928 23:47:53.142678 8713 consensus_queue.cc:260] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87e11991ecb430fb1daecce6f2a14f4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45923 } }
I20260928 23:47:53.142831 8713 raft_consensus.cc:388] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:53.142900 8713 raft_consensus.cc:482] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:53.142976 8713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:53.143899 8713 raft_consensus.cc:504] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87e11991ecb430fb1daecce6f2a14f4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45923 } }
I20260928 23:47:53.144236 8713 leader_election.cc:302] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a87e11991ecb430fb1daecce6f2a14f4; no voters:
I20260928 23:47:53.144583 8713 leader_election.cc:288] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:53.144724 8718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:53.144937 8718 raft_consensus.cc:686] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [term 1 LEADER]: Becoming Leader. State: Replica: a87e11991ecb430fb1daecce6f2a14f4, State: Running, Role: LEADER
I20260928 23:47:53.145323 8718 consensus_queue.cc:237] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87e11991ecb430fb1daecce6f2a14f4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45923 } }
I20260928 23:47:53.145946 8713 sys_catalog.cc:564] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:53.147138 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a87e11991ecb430fb1daecce6f2a14f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87e11991ecb430fb1daecce6f2a14f4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45923 } } }
I20260928 23:47:53.147260 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:53.147560 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a87e11991ecb430fb1daecce6f2a14f4. Latest consensus state: current_term: 1 leader_uuid: "a87e11991ecb430fb1daecce6f2a14f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a87e11991ecb430fb1daecce6f2a14f4" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45923 } } }
I20260928 23:47:53.147634 8727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:53.147648 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:53.150903 8727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:53.157367 8727 catalog_manager.cc:1347] Generated new cluster ID: 9ad0a6a4c433416b845f5537dc24c31f
I20260928 23:47:53.157459 8727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:53.167771 8727 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:53.169286 8727 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:53.174443 8727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4: Generated new TSK 0
I20260928 23:47:53.175148 8727 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.112.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
Entry for principal kudu/127.21.112.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab.
I20260928 23:47:53.212863 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:45923
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:53.337291 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:53.337635 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:53.337720 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:53.341883 8739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:53.341938 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:53.342022 8739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:53.347617 8739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.65.keytab
--principal=kudu/127.21.112.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:53.348805 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:53.349889 8739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:53.360342 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:53.360342 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:53.360397 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:53.360442 8739 server_base.cc:1034] running on GCE node
I20260928 23:47:53.361052 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:53.361797 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:53.362999 8739 hybrid_clock.cc:648] HybridClock initialized: now 1790639273362970 us; error 48 us; skew 500 ppm
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:53.366264 8739 init.cc:377] Logged in from keytab as kudu/127.21.112.65@KRBTEST.COM (short username kudu)
I20260928 23:47:53.367224 8739 webserver.cc:460] Webserver started at http://127.21.112.65:44261/ using document root <none> and password file <none>
I20260928 23:47:53.367823 8739 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:53.367916 8739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:53.368140 8739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:53.370353 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "2c0b56387119bb2ca352cc6d03b6253e"
server_key_iv: "82cd7bd95809456e9510652deff80cad"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.370929 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "2c0b56387119bb2ca352cc6d03b6253e"
server_key_iv: "82cd7bd95809456e9510652deff80cad"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.374727 8739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 23:47:53.377177 8754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.378588 8739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:53.378731 8739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "2c0b56387119bb2ca352cc6d03b6253e"
server_key_iv: "82cd7bd95809456e9510652deff80cad"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.378850 8739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:53.395681 8739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:53.399271 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:53.399495 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:53.400118 8739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:53.401010 8739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:53.401084 8739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.401158 8739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:53.401209 8739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.410804 8739 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:40759
I20260928 23:47:53.410840 8867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:40759 every 8 connection(s)
I20260928 23:47:53.411756 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:53.420008 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8739
I20260928 23:47:53.420192 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:47:53.420526 21953 external_mini_cluster.cc:1428] Setting key 06217c125b3391068978e647299c0f14
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.65@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:53.426766 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.413793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:41567 (local address 127.21.112.126:45923)
0928 23:47:53.413983 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.413986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.414882 (+ 896us) server_negotiation.cc:407] Connection header received
0928 23:47:53.416209 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.416212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.416260 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.416343 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:53.417517 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.418087 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.419264 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.419524 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.419612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.422531 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:53.422537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:53.422542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:53.422572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:53.424278 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.424967 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.424969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.424970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.425045 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.425464 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.425467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.425468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.425703 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:53.425784 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.426540 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.426644 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 23:47:53.428117 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.414256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:45923 (local address 127.21.112.65:41567)
0928 23:47:53.414712 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:53.414765 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:53.416013 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:53.416511 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:53.416518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:53.416895 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:53.417375 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.417389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.418205 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.418207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.419141 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.419150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.419666 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.419669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.419826 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.420469 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:53.420487 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:53.422354 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:53.424412 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:53.424420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:53.424439 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:53.424837 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:53.425137 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:53.425141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:53.425143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:53.425356 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:53.425812 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:53.425821 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:53.426266 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 23:47:53.426509 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 23:47:53.426923 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260928 23:47:53.429476 8868 heartbeater.cc:347] Connected to a master server at 127.21.112.126:45923
I20260928 23:47:53.429785 8868 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:53.430300 8868 heartbeater.cc:511] Master 127.21.112.126:45923 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.112.66@KRBTEST.COM; defaulting to no policy
I20260928 23:47:53.432013 8674 ts_manager.cc:194] Registered new tserver with Master: 4dd0bcf5d2b44c54ba86fc55a7dba002 (127.21.112.65:40759)
I20260928 23:47:53.433218 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.65@KRBTEST.COM'} at 127.21.112.65:41567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
Entry for principal kudu/127.21.112.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab.
I20260928 23:47:53.450613 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:45923
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:53.565627 8873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:53.565891 8873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:53.565943 8873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:53.570011 8873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:53.570071 8873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:53.570166 8873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:53.575524 8873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.66.keytab
--principal=kudu/127.21.112.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:53.576656 8873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:53.577827 8873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:53.588004 8878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:53.587991 8879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:53.588122 8873 server_base.cc:1034] running on GCE node
W20260928 23:47:53.587991 8881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:53.588683 8873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:53.589375 8873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:53.590580 8873 hybrid_clock.cc:648] HybridClock initialized: now 1790639273590560 us; error 39 us; skew 500 ppm
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:53.593334 8873 init.cc:377] Logged in from keytab as kudu/127.21.112.66@KRBTEST.COM (short username kudu)
I20260928 23:47:53.594357 8873 webserver.cc:460] Webserver started at http://127.21.112.66:43151/ using document root <none> and password file <none>
I20260928 23:47:53.594947 8873 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:53.595041 8873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:53.595283 8873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:53.597667 8873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "f27e191a80074dca91b94ef394aab881"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "b41671d61c7e0ffb75a0cae4b5cf4798"
server_key_iv: "c61d1754ce63bc1212ee6f3575e958d2"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.598297 8873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "f27e191a80074dca91b94ef394aab881"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "b41671d61c7e0ffb75a0cae4b5cf4798"
server_key_iv: "c61d1754ce63bc1212ee6f3575e958d2"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.602293 8873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:47:53.604868 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.606483 8873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:53.606640 8873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "f27e191a80074dca91b94ef394aab881"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "b41671d61c7e0ffb75a0cae4b5cf4798"
server_key_iv: "c61d1754ce63bc1212ee6f3575e958d2"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.606771 8873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:53.626959 8873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:53.630368 8873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:53.630580 8873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:53.631202 8873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:53.632074 8873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:53.632145 8873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.632220 8873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:53.632277 8873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.642062 8873 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:45811
I20260928 23:47:53.642083 9001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:45811 every 8 connection(s)
I20260928 23:47:53.642969 8873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:53.647563 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 8873
I20260928 23:47:53.647684 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:47:53.647993 21953 external_mini_cluster.cc:1428] Setting key 9e3c5bfc365425d15f8ae0ce9fe56db2
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.66@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:53.657750 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.644776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:57559 (local address 127.21.112.126:45923)
0928 23:47:53.644944 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.644949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.645868 (+ 919us) server_negotiation.cc:407] Connection header received
0928 23:47:53.647314 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.647320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.647380 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.647481 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:53.648972 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.649701 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.650904 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.651154 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.651271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.654056 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:53.654060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:53.654062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:53.654095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:53.655628 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.656168 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.656172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.656174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.656258 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.656604 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.656609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.656611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.656800 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:53.656910 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.657414 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.657583 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.21.112.67@KRBTEST.COM; defaulting to no policy
I20260928 23:47:53.658656 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.645189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:45923 (local address 127.21.112.66:57559)
0928 23:47:53.645723 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:53.645768 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:53.647073 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:53.647716 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:53.647727 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:53.648213 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:53.648837 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.648853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.649834 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.649837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.650767 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.650777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.651318 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.651323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.651510 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.652136 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:53.652152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:53.653868 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:53.655754 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:53.655760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:53.655777 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:53.656056 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:53.656353 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:53.656357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:53.656359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:53.656506 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:53.656907 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:53.656914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:53.657177 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 23:47:53.657372 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 23:47:53.657745 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260928 23:47:53.659936 9002 heartbeater.cc:347] Connected to a master server at 127.21.112.126:45923
I20260928 23:47:53.660187 9002 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:53.660805 9002 heartbeater.cc:511] Master 127.21.112.126:45923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.67@KRBTEST.COM" created.
I20260928 23:47:53.662053 8674 ts_manager.cc:194] Registered new tserver with Master: f27e191a80074dca91b94ef394aab881 (127.21.112.66:45811)
I20260928 23:47:53.662767 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.66@KRBTEST.COM'} at 127.21.112.66:57559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
Entry for principal kudu/127.21.112.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab.
I20260928 23:47:53.680529 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:45923
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:53.795039 9007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:53.795317 9007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:53.795388 9007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:53.799480 9007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:53.799547 9007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:53.799638 9007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:53.805194 9007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.67.keytab
--principal=kudu/127.21.112.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:45923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:53.806473 9007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:53.807528 9007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:53.817963 9012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:53.818248 9013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:53.818002 9015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:53.818024 9007 server_base.cc:1034] running on GCE node
I20260928 23:47:53.818836 9007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:53.819581 9007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:53.820784 9007 hybrid_clock.cc:648] HybridClock initialized: now 1790639273820766 us; error 43 us; skew 500 ppm
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:53.823741 9007 init.cc:377] Logged in from keytab as kudu/127.21.112.67@KRBTEST.COM (short username kudu)
I20260928 23:47:53.824767 9007 webserver.cc:460] Webserver started at http://127.21.112.67:44287/ using document root <none> and password file <none>
I20260928 23:47:53.825361 9007 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:53.825466 9007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:53.825739 9007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:53.827955 9007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "617adf276faa4c8387f2f0a2b918e2df"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "14d527cedd19aae9ef6c7f2bc3c02ad0"
server_key_iv: "af3592403185afe64336d54662b7bad7"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.828553 9007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "617adf276faa4c8387f2f0a2b918e2df"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "14d527cedd19aae9ef6c7f2bc3c02ad0"
server_key_iv: "af3592403185afe64336d54662b7bad7"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.832366 9007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:47:53.834877 9022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.836140 9007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:47:53.836319 9007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "617adf276faa4c8387f2f0a2b918e2df"
format_stamp: "Formatted at 2026-09-28 23:47:53 on dist-test-slave-c3s8"
server_key: "14d527cedd19aae9ef6c7f2bc3c02ad0"
server_key_iv: "af3592403185afe64336d54662b7bad7"
server_key_version: "encryptionkey@0"
I20260928 23:47:53.836437 9007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:53.876482 9007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:53.880362 9007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:53.880584 9007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:53.881213 9007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:53.882158 9007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:53.882230 9007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.882335 9007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:53.882368 9007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:53.892134 9007 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:46059
I20260928 23:47:53.892164 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:46059 every 8 connection(s)
I20260928 23:47:53.893069 9007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:53.897936 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9007
I20260928 23:47:53.898102 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:47:53.898484 21953 external_mini_cluster.cc:1428] Setting key 3eff0de4f73380c3c5465501e9ea00fa
Sep 28 23:47:53 dist-test-slave-c3s8 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.67@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:47:53.907646 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.895106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:46715 (local address 127.21.112.126:45923)
0928 23:47:53.895251 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.895255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.896090 (+ 835us) server_negotiation.cc:407] Connection header received
0928 23:47:53.897777 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.897784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.897855 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.897972 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:53.899310 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.899884 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.901063 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.901302 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.901419 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.904008 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:53.904012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:53.904015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:53.904041 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:53.905434 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.906019 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.906024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.906026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.906114 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.906504 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.906507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.906509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.906709 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:53.906819 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.907349 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.907482 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:47:53.908493 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.895446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:45923 (local address 127.21.112.67:46715)
0928 23:47:53.895926 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:53.895972 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:53.897463 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:53.898186 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:53.898196 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:53.898642 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:53.899151 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.899165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.900025 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.900033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.900934 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.900943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.901458 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.901462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.901701 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.902337 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:47:53.902354 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:53.903834 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:53.905613 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:53.905624 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:53.905637 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:53.905900 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:53.906212 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:47:53.906215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:47:53.906217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:47:53.906363 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:47:53.906818 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:53.906826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:53.907099 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 23:47:53.907295 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 23:47:53.907650 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260928 23:47:53.909749 9136 heartbeater.cc:347] Connected to a master server at 127.21.112.126:45923
I20260928 23:47:53.910004 9136 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:53.910602 9136 heartbeater.cc:511] Master 127.21.112.126:45923 requested a full tablet report, sending...
I20260928 23:47:53.911801 8676 ts_manager.cc:194] Registered new tserver with Master: 617adf276faa4c8387f2f0a2b918e2df (127.21.112.67:46059)
I20260928 23:47:53.912406 8676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.112.67@KRBTEST.COM'} at 127.21.112.67:46715
I20260928 23:47:53.912609 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:53.922163 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.914834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.21.112.126:45923)
0928 23:47:53.914949 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.914952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.915153 (+ 201us) server_negotiation.cc:407] Connection header received
0928 23:47:53.915301 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.915303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.915349 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.915417 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:53.916077 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.916666 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.917580 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.917792 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.917876 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.919187 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:53.919190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:53.919192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:53.919213 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:53.920536 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.920924 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.920927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.920928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.921000 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:53.921257 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:53.921259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:53.921260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:53.921499 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:53.921734 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.921957 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.922047 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:47:53.925028 8676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:53.928046 8676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:53.942972 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.936874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.21.112.67:46059)
0928 23:47:53.938285 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.938294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.938313 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:47:53.938374 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.938377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.938529 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.938861 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:53.939110 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.939855 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.941423 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.942265 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.942391 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.942556 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.942717 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.942811 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":144,"threads_started":1}
I20260928 23:47:53.942965 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.936666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:46059 (local address 127.0.0.1:52992)
0928 23:47:53.937085 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:53.937138 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:53.937283 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:53.938697 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:53.938699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:53.938711 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:53.938974 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.938987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.939991 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.939997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.941274 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.941285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.942390 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.942394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.942507 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.942544 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:47:53.942686 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 23:47:53.942811 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":65,"thread_start_us":78,"threads_started":1}
I20260928 23:47:53.943676 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.937407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40759 (local address 127.0.0.1:43124)
0928 23:47:53.937925 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:53.937940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:53.938026 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:53.938642 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:53.938651 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:53.938678 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:53.939005 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.939013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.940634 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.940637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.941998 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.942012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.943236 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.943244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.943365 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.943388 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:47:53.943442 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:47:53.943520 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":108,"thread_start_us":98,"threads_started":1}
I20260928 23:47:53.944453 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.937678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.21.112.65:40759)
0928 23:47:53.938116 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.938125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.938151 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:47:53.938241 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.938246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.938458 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.938629 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:53.939147 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.940490 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.942157 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.943050 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.943976 (+ 926us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.944133 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.944189 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.944303 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":104,"threads_started":1}
I20260928 23:47:53.945952 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.937058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:45811 (local address 127.0.0.1:43876)
0928 23:47:53.937748 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:53.937766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:53.937862 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:53.938793 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:53.938798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:53.938812 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:53.938976 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.938985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.940614 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.940618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.941900 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:53.941910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.943543 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.943546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:53.943639 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.943661 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:47:53.945727 (+ 2066us) client_negotiation.cc:240] Negotiation successful
0928 23:47:53.945826 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":80,"thread_start_us":92,"threads_started":1}
I20260928 23:47:53.946336 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:53.937475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.21.112.66:45811)
0928 23:47:53.938115 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:53.938127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:53.938151 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:47:53.938233 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:53.938238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:53.938458 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:53.938629 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:53.939153 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.940463 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.942033 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:53.943201 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:53.943321 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:53.943480 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:53.946094 (+ 2614us) server_negotiation.cc:299] Negotiation successful
0928 23:47:53.946203 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":129,"threads_started":1}
I20260928 23:47:53.946553 9070 tablet_service.cc:1467] Processing CreateTablet for tablet 148020855e9c4d7f9330126727748c45 (DEFAULT_TABLE table=test-table [id=7d5d254808bc4b589d63aa60b153d38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:53.947362 8802 tablet_service.cc:1467] Processing CreateTablet for tablet 148020855e9c4d7f9330126727748c45 (DEFAULT_TABLE table=test-table [id=7d5d254808bc4b589d63aa60b153d38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:53.947842 9070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148020855e9c4d7f9330126727748c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:53.948313 8802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148020855e9c4d7f9330126727748c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:53.949374 8935 tablet_service.cc:1467] Processing CreateTablet for tablet 148020855e9c4d7f9330126727748c45 (DEFAULT_TABLE table=test-table [id=7d5d254808bc4b589d63aa60b153d38e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:53.950453 8935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 148020855e9c4d7f9330126727748c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:53.954734 9153 tablet_bootstrap.cc:492] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: Bootstrap starting.
I20260928 23:47:53.955691 9154 tablet_bootstrap.cc:492] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Bootstrap starting.
I20260928 23:47:53.957167 9153 tablet_bootstrap.cc:654] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:53.958005 9154 tablet_bootstrap.cc:654] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:53.958094 9153 log.cc:824] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:53.958498 9155 tablet_bootstrap.cc:492] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Bootstrap starting.
I20260928 23:47:53.959394 9154 log.cc:824] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:53.961007 9153 tablet_bootstrap.cc:492] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: No bootstrap required, opened a new log
I20260928 23:47:53.961112 9154 tablet_bootstrap.cc:492] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: No bootstrap required, opened a new log
I20260928 23:47:53.961241 9153 ts_tablet_manager.cc:1397] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:47:53.961318 9154 ts_tablet_manager.cc:1397] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:47:53.961345 9155 tablet_bootstrap.cc:654] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:53.962719 9155 log.cc:824] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:53.964931 9155 tablet_bootstrap.cc:492] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: No bootstrap required, opened a new log
I20260928 23:47:53.964718 9154 raft_consensus.cc:348] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:53.965013 9154 raft_consensus.cc:374] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:53.965076 9154 raft_consensus.cc:729] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617adf276faa4c8387f2f0a2b918e2df, State: Initialized, Role: FOLLOWER
I20260928 23:47:53.965133 9155 ts_tablet_manager.cc:1397] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 23:47:53.965605 9154 consensus_queue.cc:260] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:53.966277 9136 heartbeater.cc:503] Master 127.21.112.126:45923 was elected leader, sending a full tablet report...
I20260928 23:47:53.966117 9153 raft_consensus.cc:348] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:53.966544 9153 raft_consensus.cc:374] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:53.966606 9153 raft_consensus.cc:729] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd0bcf5d2b44c54ba86fc55a7dba002, State: Initialized, Role: FOLLOWER
I20260928 23:47:53.966663 9154 ts_tablet_manager.cc:1428] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 23:47:53.967100 9153 consensus_queue.cc:260] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:53.967821 8868 heartbeater.cc:503] Master 127.21.112.126:45923 was elected leader, sending a full tablet report...
I20260928 23:47:53.968206 9153 ts_tablet_manager.cc:1428] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 23:47:53.969367 9155 raft_consensus.cc:348] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:53.969663 9155 raft_consensus.cc:374] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:53.969740 9155 raft_consensus.cc:729] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27e191a80074dca91b94ef394aab881, State: Initialized, Role: FOLLOWER
I20260928 23:47:53.970193 9155 consensus_queue.cc:260] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:53.971266 9155 ts_tablet_manager.cc:1428] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:47:53.971240 9002 heartbeater.cc:503] Master 127.21.112.126:45923 was elected leader, sending a full tablet report...
W20260928 23:47:54.144105 9003 tablet.cc:2369] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:47:54.144124 9137 tablet.cc:2369] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:54.144068 9161 raft_consensus.cc:482] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:54.144366 9161 raft_consensus.cc:504] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:54.145607 9161 leader_election.cc:288] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 617adf276faa4c8387f2f0a2b918e2df (127.21.112.67:46059), 4dd0bcf5d2b44c54ba86fc55a7dba002 (127.21.112.65:40759)
I20260928 23:47:54.149540 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.145936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:46059 (local address 127.21.112.66:47783)
0928 23:47:54.146123 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:54.146140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:54.146301 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:54.146645 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:54.146648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:54.146669 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:54.146796 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.146803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.147655 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.147658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.148459 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.148469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.149109 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.149112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.149216 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.149237 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:47:54.149332 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:47:54.149405 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 23:47:54.149567 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.145981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:47783 (local address 127.21.112.67:46059)
0928 23:47:54.146119 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:54.146126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:54.146295 (+ 169us) server_negotiation.cc:407] Connection header received
0928 23:47:54.146417 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:54.146420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:54.146500 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:54.146633 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:54.146919 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.147535 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.148583 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.148988 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.149121 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.149174 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:54.149358 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 23:47:54.149423 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:47:54.150002 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.146226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40759 (local address 127.21.112.66:54743)
0928 23:47:54.146609 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:54.146621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:54.146747 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:54.146995 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:54.146998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:54.147013 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:54.147165 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.147174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.147932 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.147937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.148899 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.148908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.149591 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.149594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.149695 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.149723 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:47:54.149809 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:47:54.149870 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":104,"threads_started":1}
I20260928 23:47:54.150178 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.146400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:54743 (local address 127.21.112.65:40759)
0928 23:47:54.146541 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:54.146545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:54.146711 (+ 166us) server_negotiation.cc:407] Connection header received
0928 23:47:54.146836 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:54.146841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:54.146885 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:54.146974 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:54.147279 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.147822 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.149028 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.149444 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.149596 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.149643 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:54.149987 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 23:47:54.150069 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:47:54.150231 9090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148020855e9c4d7f9330126727748c45" candidate_uuid: "f27e191a80074dca91b94ef394aab881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617adf276faa4c8387f2f0a2b918e2df" is_pre_election: true
I20260928 23:47:54.150547 9090 raft_consensus.cc:2454] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27e191a80074dca91b94ef394aab881 in term 0.
I20260928 23:47:54.150875 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148020855e9c4d7f9330126727748c45" candidate_uuid: "f27e191a80074dca91b94ef394aab881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" is_pre_election: true
I20260928 23:47:54.151108 8892 leader_election.cc:302] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 617adf276faa4c8387f2f0a2b918e2df, f27e191a80074dca91b94ef394aab881; no voters:
I20260928 23:47:54.151177 8822 raft_consensus.cc:2454] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27e191a80074dca91b94ef394aab881 in term 0.
I20260928 23:47:54.151399 9161 raft_consensus.cc:2781] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:54.151500 9161 raft_consensus.cc:482] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:54.151558 9161 raft_consensus.cc:3037] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:54.152635 9161 raft_consensus.cc:504] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:54.152977 9161 leader_election.cc:288] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [CANDIDATE]: Term 1 election: Requested vote from peers 617adf276faa4c8387f2f0a2b918e2df (127.21.112.67:46059), 4dd0bcf5d2b44c54ba86fc55a7dba002 (127.21.112.65:40759)
I20260928 23:47:54.153337 9090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148020855e9c4d7f9330126727748c45" candidate_uuid: "f27e191a80074dca91b94ef394aab881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617adf276faa4c8387f2f0a2b918e2df"
I20260928 23:47:54.153375 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "148020855e9c4d7f9330126727748c45" candidate_uuid: "f27e191a80074dca91b94ef394aab881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002"
I20260928 23:47:54.153466 9090 raft_consensus.cc:3037] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:54.153475 8822 raft_consensus.cc:3037] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:54.154786 8822 raft_consensus.cc:2454] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f27e191a80074dca91b94ef394aab881 in term 1.
I20260928 23:47:54.154786 9090 raft_consensus.cc:2454] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f27e191a80074dca91b94ef394aab881 in term 1.
I20260928 23:47:54.155076 8892 leader_election.cc:302] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 617adf276faa4c8387f2f0a2b918e2df, f27e191a80074dca91b94ef394aab881; no voters:
I20260928 23:47:54.155257 9161 raft_consensus.cc:2781] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:54.155488 9161 raft_consensus.cc:686] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 1 LEADER]: Becoming Leader. State: Replica: f27e191a80074dca91b94ef394aab881, State: Running, Role: LEADER
I20260928 23:47:54.155848 9161 consensus_queue.cc:237] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } }
I20260928 23:47:54.159104 8676 catalog_manager.cc:5495] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 reported cstate change: term changed from 0 to 1, leader changed from <none> to f27e191a80074dca91b94ef394aab881 (127.21.112.66). New cstate: current_term: 1 leader_uuid: "f27e191a80074dca91b94ef394aab881" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27e191a80074dca91b94ef394aab881" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 45811 } health_report { overall_health: HEALTHY } } }
W20260928 23:47:54.163094 8869 tablet.cc:2369] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:54.185742 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.181724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.21.112.66:45811)
0928 23:47:54.181848 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:54.181852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:54.181880 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:47:54.181960 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:54.181963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:54.182025 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:54.182190 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:54.182519 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.183100 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.184100 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.184342 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.184464 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.184799 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:54.185256 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:54.185408 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:54.185473 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:47:54.185561 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:47:54.192512 9090 raft_consensus.cc:1261] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 1 FOLLOWER]: Refusing update from remote peer f27e191a80074dca91b94ef394aab881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:54.192512 8822 raft_consensus.cc:1261] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 1 FOLLOWER]: Refusing update from remote peer f27e191a80074dca91b94ef394aab881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:54.193286 9161 consensus_queue.cc:1035] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "617adf276faa4c8387f2f0a2b918e2df" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 46059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:54.193497 9163 consensus_queue.cc:1035] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dd0bcf5d2b44c54ba86fc55a7dba002" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 40759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:54.202042 9167 mvcc.cc:205] Tried to move back new op lower bound from 7334458467084165120 to 7334458466945257472. Current Snapshot: MvccSnapshot[applied={T|T < 7334458467084165120}]
I20260928 23:47:54.209751 8675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:54.209939 8675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:54.212702 8675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a87e11991ecb430fb1daecce6f2a14f4: Sending DeleteTablet for 3 replicas of tablet 148020855e9c4d7f9330126727748c45
I20260928 23:47:54.213605 9070 tablet_service.cc:1514] Processing DeleteTablet for tablet 148020855e9c4d7f9330126727748c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:54 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:52992
I20260928 23:47:54.213635 8935 tablet_service.cc:1514] Processing DeleteTablet for tablet 148020855e9c4d7f9330126727748c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:54 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:43876
I20260928 23:47:54.214242 9177 tablet_replica.cc:330] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: stopping tablet replica
I20260928 23:47:54.214540 9177 raft_consensus.cc:2229] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:47:54.214525 8802 tablet_service.cc:1514] Processing DeleteTablet for tablet 148020855e9c4d7f9330126727748c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:54 UTC) from {username='kudu', principal='kudu/127.21.112.126@KRBTEST.COM'} at 127.0.0.1:43124
I20260928 23:47:54.214824 9177 raft_consensus.cc:2258] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:54.215042 9178 tablet_replica.cc:330] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002: stopping tablet replica
I20260928 23:47:54.215350 9178 raft_consensus.cc:2229] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:54.215687 9178 raft_consensus.cc:2258] T 148020855e9c4d7f9330126727748c45 P 4dd0bcf5d2b44c54ba86fc55a7dba002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:54.216117 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8739
I20260928 23:47:54.216372 9177 ts_tablet_manager.cc:1905] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:54.218731 9177 ts_tablet_manager.cc:1918] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:54.218808 9177 log.cc:1196] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-1/wal/wals/148020855e9c4d7f9330126727748c45
I20260928 23:47:54.219024 9177 ts_tablet_manager.cc:1939] T 148020855e9c4d7f9330126727748c45 P f27e191a80074dca91b94ef394aab881: Deleting consensus metadata
I20260928 23:47:54.220489 8663 catalog_manager.cc:4841] TS f27e191a80074dca91b94ef394aab881 (127.21.112.66:45811): tablet 148020855e9c4d7f9330126727748c45 (table test-table [id=7d5d254808bc4b589d63aa60b153d38e]) successfully deleted
I20260928 23:47:54.221830 9176 tablet_replica.cc:330] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: stopping tablet replica
I20260928 23:47:54.222115 9176 raft_consensus.cc:2229] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:54.222446 9176 raft_consensus.cc:2258] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:54.223897 9176 ts_tablet_manager.cc:1905] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:47:54.226002 8664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:40759 (error 108)
I20260928 23:47:54.226668 9176 ts_tablet_manager.cc:1918] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:54.226799 9176 log.cc:1196] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/148020855e9c4d7f9330126727748c45
I20260928 23:47:54.226920 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8873
I20260928 23:47:54.227176 9176 ts_tablet_manager.cc:1939] T 148020855e9c4d7f9330126727748c45 P 617adf276faa4c8387f2f0a2b918e2df: Deleting consensus metadata
I20260928 23:47:54.227416 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.227064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:40759 (local address 127.0.0.1:43132)
0928 23:47:54.227217 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:54.227306 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:40759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 23:47:54.227782 8664 catalog_manager.cc:4584] TS 4dd0bcf5d2b44c54ba86fc55a7dba002 (127.21.112.65:40759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 148020855e9c4d7f9330126727748c45: Network error: Client connection negotiation failed: client connection to 127.21.112.65:40759: connect: Connection refused (error 111)
I20260928 23:47:54.228693 8664 catalog_manager.cc:4841] TS 617adf276faa4c8387f2f0a2b918e2df (127.21.112.67:46059): tablet 148020855e9c4d7f9330126727748c45 (table test-table [id=7d5d254808bc4b589d63aa60b153d38e]) successfully deleted
I20260928 23:47:54.234588 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9007
I20260928 23:47:54.240126 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 8645
2026-09-28T23:47:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2479 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T23:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:54Z Disabled control of system clock
I20260928 23:47:54.271662 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36763
--webserver_interface=127.21.112.126
--webserver_port=0
--builtin_ntp_servers=127.21.112.84:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:47:54.384580 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:54.384878 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:54.384917 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:54.389097 9183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:54.389163 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:54.389178 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:54.389195 9183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:54.389216 9183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:54.394090 9183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:54.395159 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:54.396497 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:54.405799 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:54.405902 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:54.405884 9183 server_base.cc:1034] running on GCE node
W20260928 23:47:54.405799 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:54.406623 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:54.407891 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:54.409082 9183 hybrid_clock.cc:648] HybridClock initialized: now 1790639274409052 us; error 41 us; skew 500 ppm
I20260928 23:47:54.411281 9183 webserver.cc:460] Webserver started at http://127.21.112.126:42835/ using document root <none> and password file <none>
I20260928 23:47:54.411911 9183 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:54.411975 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:54.412294 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:54.414532 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "ca7d44dcddf8487cb192ecb524ae5d54"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "86cde08b3d6b341ea17472e08c9ce68b"
server_key_iv: "eba3fdb146c4bc60072da009997a3b0a"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.415117 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "ca7d44dcddf8487cb192ecb524ae5d54"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "86cde08b3d6b341ea17472e08c9ce68b"
server_key_iv: "eba3fdb146c4bc60072da009997a3b0a"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.418980 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:47:54.421655 9197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.422974 9183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:47:54.423135 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "ca7d44dcddf8487cb192ecb524ae5d54"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "86cde08b3d6b341ea17472e08c9ce68b"
server_key_iv: "eba3fdb146c4bc60072da009997a3b0a"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.423264 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:54.438912 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:54.439673 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:54.439882 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:54.446993 9183 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36763
I20260928 23:47:54.447017 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36763 every 8 connection(s)
I20260928 23:47:54.448021 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:54.451210 9250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:54.456844 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54: Bootstrap starting.
I20260928 23:47:54.457924 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9183
I20260928 23:47:54.458051 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:54.458346 21953 external_mini_cluster.cc:1428] Setting key ace7caa117411e348b5e58caa6b6cca1
I20260928 23:47:54.459851 9250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:54.460633 9250 log.cc:824] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:54.462900 9250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54: No bootstrap required, opened a new log
I20260928 23:47:54.467345 9250 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36763 } }
I20260928 23:47:54.467598 9250 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:54.467677 9250 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7d44dcddf8487cb192ecb524ae5d54, State: Initialized, Role: FOLLOWER
I20260928 23:47:54.468060 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.459621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.21.112.126:36763)
0928 23:47:54.460325 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:54.460339 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:54.460374 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:47:54.461100 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:54.461125 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:54.461618 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:54.461905 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:54.462252 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.463273 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.464551 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.465023 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.465137 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.466080 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:54.466086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:54.466098 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:54.466552 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:54.466606 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:54.466626 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:54.466701 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:54.466858 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 23:47:54.467160 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260928 23:47:54.468214 9250 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36763 } }
I20260928 23:47:54.468374 9250 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:54.468442 9250 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:54.468540 9250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:54.469648 9250 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36763 } }
I20260928 23:47:54.470000 9250 leader_election.cc:302] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7d44dcddf8487cb192ecb524ae5d54; no voters:
I20260928 23:47:54.470337 9250 leader_election.cc:288] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:54.470494 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:54.470767 9255 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [term 1 LEADER]: Becoming Leader. State: Replica: ca7d44dcddf8487cb192ecb524ae5d54, State: Running, Role: LEADER
I20260928 23:47:54.471064 9255 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36763 } }
I20260928 23:47:54.471539 9250 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:54.472354 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7d44dcddf8487cb192ecb524ae5d54. Latest consensus state: current_term: 1 leader_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36763 } } }
I20260928 23:47:54.472488 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:54.472752 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d44dcddf8487cb192ecb524ae5d54" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36763 } } }
I20260928 23:47:54.472877 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:54.473155 9264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:54.476229 9264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:54.481874 9264 catalog_manager.cc:1347] Generated new cluster ID: 1c96008dde6042cea10c81074d9656a1
I20260928 23:47:54.481954 9264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:54.492919 9264 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:54.494099 9264 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:54.505452 9264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54: Generated new TSK 0
I20260928 23:47:54.506215 9264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:47:54.512300 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.65:0
--local_ip_for_outbound_sockets=127.21.112.65
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36763
--builtin_ntp_servers=127.21.112.84:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:47:54.628942 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:54.629208 9274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:54.629280 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:54.633301 9274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:54.633363 9274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:54.633460 9274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.65
I20260928 23:47:54.638396 9274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.112.65
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.65
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:54.639521 9274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:54.640928 9274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:54.650888 9280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:54.650923 9274 server_base.cc:1034] running on GCE node
W20260928 23:47:54.650983 9279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:54.650897 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:54.651899 9274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:54.652647 9274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:54.653812 9274 hybrid_clock.cc:648] HybridClock initialized: now 1790639274653790 us; error 34 us; skew 500 ppm
I20260928 23:47:54.662290 9274 webserver.cc:460] Webserver started at http://127.21.112.65:38801/ using document root <none> and password file <none>
I20260928 23:47:54.662943 9274 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:54.663010 9274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:54.663261 9274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:54.665455 9274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data/instance:
uuid: "ddeaf69de32a49d39371eae5b794fa8c"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "f861dc8670bcf7c8eea4af48a8984bb4"
server_key_iv: "1b0a3830ba5e9a11730be7c446333a44"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.666116 9274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance:
uuid: "ddeaf69de32a49d39371eae5b794fa8c"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "f861dc8670bcf7c8eea4af48a8984bb4"
server_key_iv: "1b0a3830ba5e9a11730be7c446333a44"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.670128 9274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:47:54.672930 9288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.674490 9274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:47:54.674597 9274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal
uuid: "ddeaf69de32a49d39371eae5b794fa8c"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "f861dc8670bcf7c8eea4af48a8984bb4"
server_key_iv: "1b0a3830ba5e9a11730be7c446333a44"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.674746 9274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:54.693174 9274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:54.694003 9274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:54.694278 9274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:54.695132 9274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:54.696259 9274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:54.696308 9274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.696348 9274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:54.696363 9274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.705945 9274 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.65:39473
I20260928 23:47:54.705971 9401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.65:39473 every 8 connection(s)
I20260928 23:47:54.706925 9274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/data/info.pb
I20260928 23:47:54.709230 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9274
I20260928 23:47:54.709379 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-0/wal/instance
I20260928 23:47:54.709748 21953 external_mini_cluster.cc:1428] Setting key d24bf6ac5a96dde2c48e856282b2619e
I20260928 23:47:54.711844 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.66:0
--local_ip_for_outbound_sockets=127.21.112.66
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36763
--builtin_ntp_servers=127.21.112.84:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:47:54.717248 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.708866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.65:45015 (local address 127.21.112.126:36763)
0928 23:47:54.709030 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:54.709035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:54.710357 (+ 1322us) server_negotiation.cc:407] Connection header received
0928 23:47:54.711897 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:54.711903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:54.711969 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:54.712060 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:54.713103 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.713745 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.715097 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.715520 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.715659 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.716734 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:54.716738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:54.716740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:54.716752 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:54.716777 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:54.716786 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:54.716862 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:54.717036 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:47:54.717102 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 23:47:54.718281 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.709381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36763 (local address 127.21.112.65:45015)
0928 23:47:54.710198 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:54.710252 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:54.711662 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:54.712282 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:54.712289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:54.712383 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:54.712893 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.712939 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.713883 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.713889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.714961 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.714971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.715672 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.715675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.715817 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.716451 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:54.716453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:54.716494 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:54.716559 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:54.716876 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:54.716886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:54.716892 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:47:54.717023 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 23:47:54.717427 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":182,"threads_started":1}
I20260928 23:47:54.719684 9402 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36763
I20260928 23:47:54.720000 9402 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:54.720764 9402 heartbeater.cc:511] Master 127.21.112.126:36763 requested a full tablet report, sending...
I20260928 23:47:54.722714 9214 ts_manager.cc:194] Registered new tserver with Master: ddeaf69de32a49d39371eae5b794fa8c (127.21.112.65:39473)
I20260928 23:47:54.724362 9214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.65:45015
W20260928 23:47:54.827829 9405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:54.828094 9405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:54.828135 9405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:54.832178 9405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:54.832244 9405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:54.832302 9405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.66
I20260928 23:47:54.837045 9405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.112.66
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.66
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:54.838168 9405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:54.839555 9405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:54.849661 9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:54.849727 9405 server_base.cc:1034] running on GCE node
W20260928 23:47:54.849658 9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:54.849661 9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:54.850292 9405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:54.850989 9405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:54.852150 9405 hybrid_clock.cc:648] HybridClock initialized: now 1790639274852148 us; error 45 us; skew 500 ppm
I20260928 23:47:54.853991 9405 webserver.cc:460] Webserver started at http://127.21.112.66:40409/ using document root <none> and password file <none>
I20260928 23:47:54.854571 9405 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:54.854631 9405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:54.854799 9405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:54.856830 9405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data/instance:
uuid: "637a80dc89db413f8fc4821031a57ff9"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "75128f6b6fb91a78a136d37403757658"
server_key_iv: "f3fd89b94140c16e9aa06f8826b4b4c9"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.857352 9405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance:
uuid: "637a80dc89db413f8fc4821031a57ff9"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "75128f6b6fb91a78a136d37403757658"
server_key_iv: "f3fd89b94140c16e9aa06f8826b4b4c9"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.861250 9405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:47:54.863929 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.865314 9405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:47:54.865581 9405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal
uuid: "637a80dc89db413f8fc4821031a57ff9"
format_stamp: "Formatted at 2026-09-28 23:47:54 on dist-test-slave-c3s8"
server_key: "75128f6b6fb91a78a136d37403757658"
server_key_iv: "f3fd89b94140c16e9aa06f8826b4b4c9"
server_key_version: "encryptionkey@0"
I20260928 23:47:54.865717 9405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:54.879374 9405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:54.880095 9405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:54.880302 9405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:54.880937 9405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:54.882036 9405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:54.882085 9405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.882160 9405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:54.882195 9405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:54.891515 9405 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.66:38513
I20260928 23:47:54.891543 9532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.66:38513 every 8 connection(s)
I20260928 23:47:54.892458 9405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/data/info.pb
I20260928 23:47:54.898751 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9405
I20260928 23:47:54.898910 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-1/wal/instance
I20260928 23:47:54.899200 21953 external_mini_cluster.cc:1428] Setting key 5f38a541459330528b1cf95e295f5c72
I20260928 23:47:54.901422 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.112.67:0
--local_ip_for_outbound_sockets=127.21.112.67
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36763
--builtin_ntp_servers=127.21.112.84:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:47:54.902380 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.894467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.66:50597 (local address 127.21.112.126:36763)
0928 23:47:54.894606 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:54.894609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:54.895686 (+ 1077us) server_negotiation.cc:407] Connection header received
0928 23:47:54.897048 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:54.897051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:54.897099 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:54.897173 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:54.898268 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.898786 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.900225 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.900453 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.900570 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.901786 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:54.901792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:54.901798 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:54.901815 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:54.901840 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:54.901854 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:54.901937 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:54.902098 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 23:47:54.902172 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:47:54.903160 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:54.894813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36763 (local address 127.21.112.66:50597)
0928 23:47:54.895535 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:54.895575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:54.896854 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:54.897327 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:54.897334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:54.897420 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:54.898126 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.898140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.898927 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.898931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.900094 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:54.900104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:54.900610 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:54.900614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:54.900781 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:54.901448 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:54.901450 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:54.901492 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:54.901610 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:54.901949 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:54.901957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:54.901962 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:47:54.902065 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 23:47:54.902299 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260928 23:47:54.904357 9533 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36763
I20260928 23:47:54.904603 9533 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:54.905208 9533 heartbeater.cc:511] Master 127.21.112.126:36763 requested a full tablet report, sending...
I20260928 23:47:54.906507 9214 ts_manager.cc:194] Registered new tserver with Master: 637a80dc89db413f8fc4821031a57ff9 (127.21.112.66:38513)
I20260928 23:47:54.907088 9214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.66:50597
W20260928 23:47:55.017841 9536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:55.018114 9536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:55.018153 9536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:55.022302 9536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:55.022367 9536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:55.022454 9536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.112.67
I20260928 23:47:55.027644 9536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.112.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.112.67
--webserver_port=0
--tserver_master_addrs=127.21.112.126:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.112.67
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:55.028891 9536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:55.030442 9536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:55.040736 9544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:55.040849 9542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:55.040736 9541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:55.040994 9536 server_base.cc:1034] running on GCE node
I20260928 23:47:55.041463 9536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:55.042201 9536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:55.043366 9536 hybrid_clock.cc:648] HybridClock initialized: now 1790639275043337 us; error 42 us; skew 500 ppm
I20260928 23:47:55.045339 9536 webserver.cc:460] Webserver started at http://127.21.112.67:40485/ using document root <none> and password file <none>
I20260928 23:47:55.046072 9536 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:55.046176 9536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:55.046445 9536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:55.048619 9536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data/instance:
uuid: "8c5bdd4dd6424e0e832da053aad6a23c"
format_stamp: "Formatted at 2026-09-28 23:47:55 on dist-test-slave-c3s8"
server_key: "505b6f3d67ad1b2c6573a4be90fb4c7b"
server_key_iv: "8957fae83bb2a30c01c508338e43e8c6"
server_key_version: "encryptionkey@0"
I20260928 23:47:55.049219 9536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance:
uuid: "8c5bdd4dd6424e0e832da053aad6a23c"
format_stamp: "Formatted at 2026-09-28 23:47:55 on dist-test-slave-c3s8"
server_key: "505b6f3d67ad1b2c6573a4be90fb4c7b"
server_key_iv: "8957fae83bb2a30c01c508338e43e8c6"
server_key_version: "encryptionkey@0"
I20260928 23:47:55.053136 9536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:47:55.055855 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:55.057186 9536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:55.057331 9536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal
uuid: "8c5bdd4dd6424e0e832da053aad6a23c"
format_stamp: "Formatted at 2026-09-28 23:47:55 on dist-test-slave-c3s8"
server_key: "505b6f3d67ad1b2c6573a4be90fb4c7b"
server_key_iv: "8957fae83bb2a30c01c508338e43e8c6"
server_key_version: "encryptionkey@0"
I20260928 23:47:55.057459 9536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:55.076193 9536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:55.076969 9536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:55.077180 9536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:55.077837 9536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:47:55.078908 9536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:47:55.078982 9536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:55.079054 9536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:47:55.079106 9536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:55.088481 9536 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.67:40627
I20260928 23:47:55.088541 9663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.67:40627 every 8 connection(s)
I20260928 23:47:55.089646 9536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/data/info.pb
I20260928 23:47:55.098927 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.091627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:39479 (local address 127.21.112.126:36763)
0928 23:47:55.091791 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.091795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.092802 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 23:47:55.094280 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.094283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.094337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.094415 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:55.095343 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.095876 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.097046 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.097263 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.097341 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.098451 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:55.098454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:55.098456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:55.098470 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:55.098489 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:55.098498 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:55.098559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.098745 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.098801 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:47:55.099319 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9536
I20260928 23:47:55.099449 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal/instance
I20260928 23:47:55.099748 21953 external_mini_cluster.cc:1428] Setting key 7a7145174d8731064f598e94bad16651
I20260928 23:47:55.100119 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.091960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:36763 (local address 127.21.112.67:39479)
0928 23:47:55.092648 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.092689 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:55.094064 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:55.094586 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:55.094593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:55.094682 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:47:55.095208 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.095221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.095985 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.095988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.096931 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.096941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.097406 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.097409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.097578 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.098171 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:47:55.098172 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:47:55.098215 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:47:55.098290 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:47:55.098606 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:47:55.098614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:47:55.098619 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:47:55.098727 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 23:47:55.099035 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260928 23:47:55.101307 9664 heartbeater.cc:347] Connected to a master server at 127.21.112.126:36763
I20260928 23:47:55.101621 9664 heartbeater.cc:464] Registering TS with master...
I20260928 23:47:55.102263 9664 heartbeater.cc:511] Master 127.21.112.126:36763 requested a full tablet report, sending...
I20260928 23:47:55.103310 9213 ts_manager.cc:194] Registered new tserver with Master: 8c5bdd4dd6424e0e832da053aad6a23c (127.21.112.67:40627)
I20260928 23:47:55.103914 9213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.112.67:39479
I20260928 23:47:55.114400 21953 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:47:55.120507 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.116442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.21.112.126:36763)
0928 23:47:55.116610 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.116615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.116757 (+ 142us) server_negotiation.cc:407] Connection header received
0928 23:47:55.116943 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.116946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.116990 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.117107 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:55.117404 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.118006 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.118889 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.119121 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.119233 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.120078 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:55.120082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:55.120084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:55.120104 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:55.120121 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:55.120130 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:55.120207 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.120303 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.120359 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 23:47:55.123570 9213 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:47:55.126695 9213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:47:55.141597 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.135626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.21.112.66:38513)
0928 23:47:55.136527 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.136541 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.136570 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:47:55.136652 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.136657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.136894 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.137086 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:55.137380 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.138530 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.140031 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.140858 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.140978 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.141169 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.141308 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.141390 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":110,"threads_started":1}
I20260928 23:47:55.141597 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.135517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38513 (local address 127.0.0.1:33250)
0928 23:47:55.136256 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.136311 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:55.136454 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:55.137092 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:55.137097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:55.137118 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:55.137263 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.137269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.138657 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.138660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.139900 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.139909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.140994 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.140997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.141118 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.141155 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:47:55.141274 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 23:47:55.141396 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":95,"threads_started":1}
I20260928 23:47:55.141880 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.136445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.21.112.65:39473)
0928 23:47:55.136783 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.136796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.137130 (+ 334us) server_negotiation.cc:407] Connection header received
0928 23:47:55.137215 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.137220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.137409 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.137590 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:55.138026 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.139131 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.140428 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.140985 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.141090 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.141275 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.141585 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.141703 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":127,"threads_started":1}
I20260928 23:47:55.142042 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.136251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:39473 (local address 127.0.0.1:54666)
0928 23:47:55.136926 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.136940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:55.137036 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:55.137773 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:55.137776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:55.137796 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:55.137884 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.137890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.139273 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.139276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.140314 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.140321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.141332 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.141336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.141415 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.141435 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:47:55.141508 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:47:55.141588 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":89,"threads_started":1}
I20260928 23:47:55.143095 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.135573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.67:40627 (local address 127.0.0.1:40222)
0928 23:47:55.136819 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.136832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:55.136917 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:55.137574 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:55.137577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:55.137595 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:55.137685 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.137691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.138772 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.138775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.140034 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.140044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.141666 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.141669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.142808 (+ 1139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.142832 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:47:55.142911 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:47:55.142978 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":146,"threads_started":1}
I20260928 23:47:55.143492 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.135779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.21.112.67:40627)
0928 23:47:55.137134 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.137146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.137167 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:47:55.137230 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.137234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.137408 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.137592 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:55.137804 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.138650 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.140160 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.140867 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.140982 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.141169 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.142966 (+ 1797us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.143071 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":90,"threads_started":1}
I20260928 23:47:55.144770 9467 tablet_service.cc:1467] Processing CreateTablet for tablet 373e3a76c54c471da4f67336b725d23a (DEFAULT_TABLE table=test-table [id=8ddcc8a634414e06843d0979d2b609ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:55.144789 9336 tablet_service.cc:1467] Processing CreateTablet for tablet 373e3a76c54c471da4f67336b725d23a (DEFAULT_TABLE table=test-table [id=8ddcc8a634414e06843d0979d2b609ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:55.145792 9467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373e3a76c54c471da4f67336b725d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:55.145833 9336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373e3a76c54c471da4f67336b725d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:55.146237 9598 tablet_service.cc:1467] Processing CreateTablet for tablet 373e3a76c54c471da4f67336b725d23a (DEFAULT_TABLE table=test-table [id=8ddcc8a634414e06843d0979d2b609ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:47:55.147214 9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373e3a76c54c471da4f67336b725d23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:55.151690 9681 tablet_bootstrap.cc:492] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: Bootstrap starting.
I20260928 23:47:55.153765 9682 tablet_bootstrap.cc:492] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: Bootstrap starting.
I20260928 23:47:55.154369 9681 tablet_bootstrap.cc:654] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:55.155104 9681 log.cc:824] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:55.155745 9682 tablet_bootstrap.cc:654] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:55.155750 9683 tablet_bootstrap.cc:492] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Bootstrap starting.
I20260928 23:47:55.156579 9682 log.cc:824] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:55.157219 9681 tablet_bootstrap.cc:492] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: No bootstrap required, opened a new log
I20260928 23:47:55.157456 9681 ts_tablet_manager.cc:1397] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 23:47:55.157544 9683 tablet_bootstrap.cc:654] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:55.158370 9683 log.cc:824] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:55.158663 9682 tablet_bootstrap.cc:492] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: No bootstrap required, opened a new log
I20260928 23:47:55.158850 9682 ts_tablet_manager.cc:1397] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 23:47:55.160130 9683 tablet_bootstrap.cc:492] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: No bootstrap required, opened a new log
I20260928 23:47:55.160369 9683 ts_tablet_manager.cc:1397] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 23:47:55.162002 9681 raft_consensus.cc:348] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.162305 9681 raft_consensus.cc:374] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:55.162362 9681 raft_consensus.cc:729] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637a80dc89db413f8fc4821031a57ff9, State: Initialized, Role: FOLLOWER
I20260928 23:47:55.162827 9681 consensus_queue.cc:260] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.163337 9682 raft_consensus.cc:348] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.163632 9682 raft_consensus.cc:374] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:55.163681 9682 raft_consensus.cc:729] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddeaf69de32a49d39371eae5b794fa8c, State: Initialized, Role: FOLLOWER
I20260928 23:47:55.163877 9681 ts_tablet_manager.cc:1428] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:47:55.164108 9682 consensus_queue.cc:260] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.164247 9533 heartbeater.cc:503] Master 127.21.112.126:36763 was elected leader, sending a full tablet report...
I20260928 23:47:55.164626 9683 raft_consensus.cc:348] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.164848 9683 raft_consensus.cc:374] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:55.164902 9683 raft_consensus.cc:729] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5bdd4dd6424e0e832da053aad6a23c, State: Initialized, Role: FOLLOWER
I20260928 23:47:55.164949 9682 ts_tablet_manager.cc:1428] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 23:47:55.165279 9402 heartbeater.cc:503] Master 127.21.112.126:36763 was elected leader, sending a full tablet report...
I20260928 23:47:55.165299 9683 consensus_queue.cc:260] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.167902 9683 ts_tablet_manager.cc:1428] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260928 23:47:55.169085 9664 heartbeater.cc:503] Master 127.21.112.126:36763 was elected leader, sending a full tablet report...
W20260928 23:47:55.208145 9403 tablet.cc:2369] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:47:55.259800 9689 raft_consensus.cc:482] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:47:55.260017 9689 raft_consensus.cc:504] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.261281 9689 leader_election.cc:288] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 637a80dc89db413f8fc4821031a57ff9 (127.21.112.66:38513), ddeaf69de32a49d39371eae5b794fa8c (127.21.112.65:39473)
I20260928 23:47:55.265479 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.261651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38513 (local address 127.21.112.67:38861)
0928 23:47:55.261777 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.261811 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:55.261951 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:55.262298 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:55.262303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:55.262321 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:55.262473 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.262484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.263595 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.263600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.264404 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.264412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.265054 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.265058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.265155 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.265176 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:47:55.265275 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:47:55.265347 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:47:55.265479 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.261726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:38861 (local address 127.21.112.66:38513)
0928 23:47:55.261909 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.261915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.261947 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:47:55.262080 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.262083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.262133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.262200 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:55.262586 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.263473 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.264526 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.264934 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.265065 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.265117 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.265289 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.265360 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:47:55.265758 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.261970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:39473 (local address 127.21.112.67:43951)
0928 23:47:55.262385 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.262396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:47:55.262534 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:47:55.262848 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:47:55.262851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:47:55.262868 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:47:55.263009 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.263020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.263822 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.263825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.264596 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:47:55.264605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.265276 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.265280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:47:55.265372 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.265392 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 23:47:55.265497 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:47:55.265584 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":121,"threads_started":1}
I20260928 23:47:55.265888 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.262093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.112.67:43951 (local address 127.21.112.65:39473)
0928 23:47:55.262223 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.262227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.262498 (+ 271us) server_negotiation.cc:407] Connection header received
0928 23:47:55.262700 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.262703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.262756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.262834 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:47:55.263160 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.263697 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.264724 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.265126 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.265276 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.265324 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.265513 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.265767 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 23:47:55.266194 9487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e3a76c54c471da4f67336b725d23a" candidate_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637a80dc89db413f8fc4821031a57ff9" is_pre_election: true
I20260928 23:47:55.266389 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e3a76c54c471da4f67336b725d23a" candidate_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddeaf69de32a49d39371eae5b794fa8c" is_pre_election: true
I20260928 23:47:55.266664 9487 raft_consensus.cc:2454] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5bdd4dd6424e0e832da053aad6a23c in term 0.
I20260928 23:47:55.266708 9356 raft_consensus.cc:2454] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c5bdd4dd6424e0e832da053aad6a23c in term 0.
I20260928 23:47:55.267189 9554 leader_election.cc:302] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637a80dc89db413f8fc4821031a57ff9, 8c5bdd4dd6424e0e832da053aad6a23c; no voters:
I20260928 23:47:55.267490 9689 raft_consensus.cc:2781] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:47:55.267568 9689 raft_consensus.cc:482] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:47:55.267624 9689 raft_consensus.cc:3037] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:55.268643 9689 raft_consensus.cc:504] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.268992 9689 leader_election.cc:288] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [CANDIDATE]: Term 1 election: Requested vote from peers 637a80dc89db413f8fc4821031a57ff9 (127.21.112.66:38513), ddeaf69de32a49d39371eae5b794fa8c (127.21.112.65:39473)
I20260928 23:47:55.269367 9487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e3a76c54c471da4f67336b725d23a" candidate_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "637a80dc89db413f8fc4821031a57ff9"
I20260928 23:47:55.269562 9487 raft_consensus.cc:3037] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:55.269599 9356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373e3a76c54c471da4f67336b725d23a" candidate_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddeaf69de32a49d39371eae5b794fa8c"
I20260928 23:47:55.269730 9356 raft_consensus.cc:3037] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:55.270728 9487 raft_consensus.cc:2454] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5bdd4dd6424e0e832da053aad6a23c in term 1.
I20260928 23:47:55.270814 9356 raft_consensus.cc:2454] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5bdd4dd6424e0e832da053aad6a23c in term 1.
I20260928 23:47:55.271157 9554 leader_election.cc:302] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 637a80dc89db413f8fc4821031a57ff9, 8c5bdd4dd6424e0e832da053aad6a23c; no voters:
I20260928 23:47:55.271358 9689 raft_consensus.cc:2781] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:55.271618 9689 raft_consensus.cc:686] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 1 LEADER]: Becoming Leader. State: Replica: 8c5bdd4dd6424e0e832da053aad6a23c, State: Running, Role: LEADER
I20260928 23:47:55.272013 9689 consensus_queue.cc:237] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } }
I20260928 23:47:55.275432 9213 catalog_manager.cc:5495] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c5bdd4dd6424e0e832da053aad6a23c (127.21.112.67). New cstate: current_term: 1 leader_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c5bdd4dd6424e0e832da053aad6a23c" member_type: VOTER last_known_addr { host: "127.21.112.67" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20260928 23:47:55.301041 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.297480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.21.112.67:40627)
0928 23:47:55.297640 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:55.297644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:55.297658 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:47:55.297734 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:55.297736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:55.297793 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:55.297883 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:47:55.298105 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.298611 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.299469 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:55.299711 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:55.299820 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:55.300105 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:47:55.300577 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:47:55.300687 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:55.300742 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:47:55.300800 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 23:47:55.307320 9356 raft_consensus.cc:1261] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c [term 1 FOLLOWER]: Refusing update from remote peer 8c5bdd4dd6424e0e832da053aad6a23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:55.307334 9487 raft_consensus.cc:1261] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8c5bdd4dd6424e0e832da053aad6a23c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:47:55.308079 9691 consensus_queue.cc:1035] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "637a80dc89db413f8fc4821031a57ff9" member_type: VOTER last_known_addr { host: "127.21.112.66" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:55.308279 9689 consensus_queue.cc:1035] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddeaf69de32a49d39371eae5b794fa8c" member_type: VOTER last_known_addr { host: "127.21.112.65" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:47:55.316660 9694 mvcc.cc:205] Tried to move back new op lower bound from 7334458471650721792 to 7334458471517319168. Current Snapshot: MvccSnapshot[applied={T|T < 7334458471650721792}]
I20260928 23:47:55.324460 9213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:55.324646 9213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:47:55.327252 9213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca7d44dcddf8487cb192ecb524ae5d54: Sending DeleteTablet for 3 replicas of tablet 373e3a76c54c471da4f67336b725d23a
I20260928 23:47:55.328096 9598 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e3a76c54c471da4f67336b725d23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:55 UTC) from {username='slave'} at 127.0.0.1:40222
I20260928 23:47:55.328096 9467 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e3a76c54c471da4f67336b725d23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:55 UTC) from {username='slave'} at 127.0.0.1:33250
I20260928 23:47:55.328655 9704 tablet_replica.cc:330] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: stopping tablet replica
I20260928 23:47:55.328662 9705 tablet_replica.cc:330] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: stopping tablet replica
I20260928 23:47:55.328987 9705 raft_consensus.cc:2229] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:47:55.329027 9704 raft_consensus.cc:2229] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:47:55.329110 9336 tablet_service.cc:1514] Processing DeleteTablet for tablet 373e3a76c54c471da4f67336b725d23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:47:55 UTC) from {username='slave'} at 127.0.0.1:54666
I20260928 23:47:55.329349 9705 raft_consensus.cc:2258] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:55.329602 9704 raft_consensus.cc:2258] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:47:55.330113 9706 tablet_replica.cc:330] T 373e3a76c54c471da4f67336b725d23a P ddeaf69de32a49d39371eae5b794fa8c: stopping tablet replica
I20260928 23:47:55.330206 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9274
I20260928 23:47:55.331506 9705 ts_tablet_manager.cc:1905] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:47:55.333750 9705 ts_tablet_manager.cc:1918] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:47:55.333859 9705 log.cc:1196] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Deleting WAL directory at /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790639014951155-21953-0/minicluster-data/ts-2/wal/wals/373e3a76c54c471da4f67336b725d23a
I20260928 23:47:55.334144 9705 ts_tablet_manager.cc:1939] T 373e3a76c54c471da4f67336b725d23a P 8c5bdd4dd6424e0e832da053aad6a23c: Deleting consensus metadata
I20260928 23:47:55.335282 9200 catalog_manager.cc:4841] TS 8c5bdd4dd6424e0e832da053aad6a23c (127.21.112.67:40627): tablet 373e3a76c54c471da4f67336b725d23a (table test-table [id=8ddcc8a634414e06843d0979d2b609ca]) successfully deleted
I20260928 23:47:55.337380 9704 ts_tablet_manager.cc:1905] T 373e3a76c54c471da4f67336b725d23a P 637a80dc89db413f8fc4821031a57ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:47:55.337565 9201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.112.65:39473 (error 108)
I20260928 23:47:55.338642 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.338321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.65:39473 (local address 127.0.0.1:54678)
0928 23:47:55.338471 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.338550 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.65:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 23:47:55.338680 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9405
W20260928 23:47:55.338941 9201 catalog_manager.cc:4584] TS ddeaf69de32a49d39371eae5b794fa8c (127.21.112.65:39473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 373e3a76c54c471da4f67336b725d23a: Network error: Client connection negotiation failed: client connection to 127.21.112.65:39473: connect: Connection refused (error 111)
I20260928 23:47:55.344081 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9536
I20260928 23:47:55.345072 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:55.344154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.66:38513 (local address 127.0.0.1:33252)
0928 23:47:55.344954 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 23:47:55.345003 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.66:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":718}
I20260928 23:47:55.349822 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9183
2026-09-28T23:47:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1108 ms)
[----------] 6 tests from EncryptionPolicyTest (10824 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:55 dist-test-slave-c3s8 krb5kdc[9710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:55 dist-test-slave-c3s8 krb5kdc[9710](info): set up 2 sockets
Sep 28 23:47:55 dist-test-slave-c3s8 krb5kdc[9710](info): commencing operation
krb5kdc: starting...
W20260928 23:47:56.394064 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:47:56 dist-test-slave-c3s8 krb5kdc[9710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:47:56.526913 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:56.637847 9723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:56.638171 9723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:56.638260 9723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:56.642340 9723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:56.642427 9723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:56.642467 9723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:56.642525 9723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:56.642563 9723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:56.647819 9723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:56.648885 9723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:56.649928 9723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:56.658627 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:56.658614 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:56.658671 9723 server_base.cc:1034] running on GCE node
W20260928 23:47:56.658614 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:56.659404 9723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:56.660570 9723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:56.661734 9723 hybrid_clock.cc:648] HybridClock initialized: now 1790639276661721 us; error 24 us; skew 500 ppm
Sep 28 23:47:56 dist-test-slave-c3s8 krb5kdc[9710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:56.664556 9723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:47:56.665495 9723 webserver.cc:460] Webserver started at http://127.0.0.1:37215/ using document root <none> and password file <none>
I20260928 23:47:56.666134 9723 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:56.666196 9723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:56.666409 9723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:56.668411 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "965854ebfd8b49ddbe47479f9735c1a1"
format_stamp: "Formatted at 2026-09-28 23:47:56 on dist-test-slave-c3s8"
server_key: "f4cc1972645097cbc2112ce57f7caf46"
server_key_iv: "5154a8c28285f6c81dc02942b9c90188"
server_key_version: "encryptionkey@0"
I20260928 23:47:56.668967 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "965854ebfd8b49ddbe47479f9735c1a1"
format_stamp: "Formatted at 2026-09-28 23:47:56 on dist-test-slave-c3s8"
server_key: "f4cc1972645097cbc2112ce57f7caf46"
server_key_iv: "5154a8c28285f6c81dc02942b9c90188"
server_key_version: "encryptionkey@0"
I20260928 23:47:56.672582 9723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 23:47:56.675076 9738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:56.676303 9723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:47:56.676481 9723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "965854ebfd8b49ddbe47479f9735c1a1"
format_stamp: "Formatted at 2026-09-28 23:47:56 on dist-test-slave-c3s8"
server_key: "f4cc1972645097cbc2112ce57f7caf46"
server_key_iv: "5154a8c28285f6c81dc02942b9c90188"
server_key_version: "encryptionkey@0"
I20260928 23:47:56.676594 9723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:56.695080 9723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:56.698483 9723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:56.698693 9723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:56.706363 9723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36835
I20260928 23:47:56.706389 9790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36835 every 8 connection(s)
I20260928 23:47:56.707312 9723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:56.710143 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:56.713651 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9723
I20260928 23:47:56.713781 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:56.714052 21953 external_mini_cluster.cc:1428] Setting key dee633584e7abde1e83b06cf5556856c
I20260928 23:47:56.716133 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1: Bootstrap starting.
I20260928 23:47:56.718559 9791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:56.719343 9791 log.cc:824] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:56.721253 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1: No bootstrap required, opened a new log
Sep 28 23:47:56 dist-test-slave-c3s8 krb5kdc[9710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:47:56.725291 9791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965854ebfd8b49ddbe47479f9735c1a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 23:47:56.725566 9791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:56.725641 9791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965854ebfd8b49ddbe47479f9735c1a1, State: Initialized, Role: FOLLOWER
I20260928 23:47:56.726125 9791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965854ebfd8b49ddbe47479f9735c1a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 23:47:56.726281 9791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:56.726377 9791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:56.726497 9791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:56.727419 9791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965854ebfd8b49ddbe47479f9735c1a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 23:47:56.727775 9791 leader_election.cc:302] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965854ebfd8b49ddbe47479f9735c1a1; no voters:
I20260928 23:47:56.728125 9791 leader_election.cc:288] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:56.728217 9796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:56.728392 9796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [term 1 LEADER]: Becoming Leader. State: Replica: 965854ebfd8b49ddbe47479f9735c1a1, State: Running, Role: LEADER
I20260928 23:47:56.728715 9796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965854ebfd8b49ddbe47479f9735c1a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 23:47:56.729151 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:56.715550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56191 (local address 127.0.0.1:36835)
0928 23:47:56.716007 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:56.716022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:56.716060 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:47:56.716801 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:56.716819 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:56.717178 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:56.717649 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:56.718506 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:56.719670 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:56.720545 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:56.721015 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:56.721198 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:56.723807 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:56.723812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:56.723826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:56.723855 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:56.725683 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:56.726246 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:56.726252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:56.726258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:56.726447 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:56.726850 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:56.726854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:56.726856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:56.727263 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:56.727511 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:56.727844 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 23:47:56.728092 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260928 23:47:56.729451 9791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:56.729962 9796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965854ebfd8b49ddbe47479f9735c1a1. Latest consensus state: current_term: 1 leader_uuid: "965854ebfd8b49ddbe47479f9735c1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965854ebfd8b49ddbe47479f9735c1a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } } }
I20260928 23:47:56.730080 9796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:56.730489 9797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "965854ebfd8b49ddbe47479f9735c1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965854ebfd8b49ddbe47479f9735c1a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } } }
I20260928 23:47:56.730618 9797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:56.732613 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 23:47:56.735590 9812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:47:56.735672 9812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:47:56.735740 9800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:56.736397 9800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:56.742251 9800 catalog_manager.cc:1347] Generated new cluster ID: f3119bf4c4f747f6997b8347e080da6d
I20260928 23:47:56.742326 9800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:56.751080 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:56.734785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40259 (local address 127.0.0.1:36835)
0928 23:47:56.734918 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:56.734923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:56.738237 (+ 3314us) server_negotiation.cc:407] Connection header received
0928 23:47:56.738407 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:56.738411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:56.738465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:56.738553 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:56.739347 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:56.739875 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:56.740941 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:56.741936 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:56.742058 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:56.747371 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:56.747375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:56.747379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:47:56.747409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:56.749097 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:56.749726 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:56.749730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:56.749732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:56.749821 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:47:56.750159 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:47:56.750163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:47:56.750164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:47:56.750391 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:56.750537 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:56.750811 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 23:47:56.750911 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:47:56.783874 9800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:56.785085 9800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:56.790819 9800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 965854ebfd8b49ddbe47479f9735c1a1: Generated new TSK 0
I20260928 23:47:56.791604 9800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:47:56.796011 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9723
W20260928 23:47:56.801376 9708 connection.cc:535] client connection to 127.0.0.1:36835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36835): Connection reset by peer (error 104)
2026-09-28T23:47:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:56 dist-test-slave-c3s8 krb5kdc[9822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:56 dist-test-slave-c3s8 krb5kdc[9822](info): set up 2 sockets
Sep 28 23:47:56 dist-test-slave-c3s8 krb5kdc[9822](info): commencing operation
krb5kdc: starting...
W20260928 23:47:57.854224 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:47:57 dist-test-slave-c3s8 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:47:57.981640 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:58.097050 9835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:58.097329 9835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:58.097374 9835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:58.101466 9835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:58.101605 9835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:58.101637 9835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:58.101656 9835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:58.101672 9835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:58.107218 9835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:58.108327 9835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:58.109355 9835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:58.118484 9843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:58.118499 9841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:58.118572 9835 server_base.cc:1034] running on GCE node
W20260928 23:47:58.118494 9840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:58.119282 9835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:58.120431 9835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:58.121608 9835 hybrid_clock.cc:648] HybridClock initialized: now 1790639278121576 us; error 47 us; skew 500 ppm
Sep 28 23:47:58 dist-test-slave-c3s8 krb5kdc[9822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:58.124850 9835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:47:58.125946 9835 webserver.cc:460] Webserver started at http://127.0.0.1:37195/ using document root <none> and password file <none>
I20260928 23:47:58.126529 9835 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:58.126622 9835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:58.126863 9835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:58.129019 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "2190d02c449f4fc08126768d1403efbe"
format_stamp: "Formatted at 2026-09-28 23:47:58 on dist-test-slave-c3s8"
server_key: "574b962e242cfeef9ad95eb7ad5e451e"
server_key_iv: "115f40c2e27c75d6ee07d31b75a1e6d2"
server_key_version: "encryptionkey@0"
I20260928 23:47:58.129647 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "2190d02c449f4fc08126768d1403efbe"
format_stamp: "Formatted at 2026-09-28 23:47:58 on dist-test-slave-c3s8"
server_key: "574b962e242cfeef9ad95eb7ad5e451e"
server_key_iv: "115f40c2e27c75d6ee07d31b75a1e6d2"
server_key_version: "encryptionkey@0"
I20260928 23:47:58.133455 9835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:47:58.136086 9850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:58.137437 9835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:58.137619 9835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "2190d02c449f4fc08126768d1403efbe"
format_stamp: "Formatted at 2026-09-28 23:47:58 on dist-test-slave-c3s8"
server_key: "574b962e242cfeef9ad95eb7ad5e451e"
server_key_iv: "115f40c2e27c75d6ee07d31b75a1e6d2"
server_key_version: "encryptionkey@0"
I20260928 23:47:58.137741 9835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:58.168597 9835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:58.172540 9835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:58.172794 9835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:58.181134 9835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33891
I20260928 23:47:58.181156 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33891 every 8 connection(s)
I20260928 23:47:58.182317 9835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:58.185243 9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:58.188884 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9835
I20260928 23:47:58.189051 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:58.189345 21953 external_mini_cluster.cc:1428] Setting key 7d61bc040e06d4c5b0f3749d87746f34
I20260928 23:47:58.191427 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe: Bootstrap starting.
I20260928 23:47:58.193794 9903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:58.194592 9903 log.cc:824] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:58.196636 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe: No bootstrap required, opened a new log
I20260928 23:47:58.198962 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:58.190866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60257 (local address 127.0.0.1:33891)
0928 23:47:58.191411 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:58.191428 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:58.191470 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:47:58.192186 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:58.192203 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:58.192576 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:58.192939 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:58.193390 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:58.194562 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:58.195533 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:58.196023 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:58.196196 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:58.197178 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:58.197183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:58.197197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:58.197675 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:58.197736 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:58.197760 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:58.197905 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:58.198084 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:47:58.198226 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":143,"threads_started":1}
I20260928 23:47:58.200484 9903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190d02c449f4fc08126768d1403efbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33891 } }
I20260928 23:47:58.200752 9903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:58.200809 9903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2190d02c449f4fc08126768d1403efbe, State: Initialized, Role: FOLLOWER
I20260928 23:47:58.201309 9903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190d02c449f4fc08126768d1403efbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33891 } }
I20260928 23:47:58.201495 9903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:58.201586 9903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:58.201704 9903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:58.202303 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:47:58.202862 9903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190d02c449f4fc08126768d1403efbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33891 } }
I20260928 23:47:58.203253 9903 leader_election.cc:302] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2190d02c449f4fc08126768d1403efbe; no voters:
I20260928 23:47:58.203612 9903 leader_election.cc:288] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:58.203795 9908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:58.204087 9908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [term 1 LEADER]: Becoming Leader. State: Replica: 2190d02c449f4fc08126768d1403efbe, State: Running, Role: LEADER
I20260928 23:47:58.204491 9908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190d02c449f4fc08126768d1403efbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33891 } }
I20260928 23:47:58.205060 9903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:58.205937 9914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2190d02c449f4fc08126768d1403efbe. Latest consensus state: current_term: 1 leader_uuid: "2190d02c449f4fc08126768d1403efbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190d02c449f4fc08126768d1403efbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33891 } } }
I20260928 23:47:58.206061 9914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:58.206429 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2190d02c449f4fc08126768d1403efbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2190d02c449f4fc08126768d1403efbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33891 } } }
I20260928 23:47:58.206486 9922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:58.206540 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:58.209496 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:58.204571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56567 (local address 127.0.0.1:33891)
0928 23:47:58.204674 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:58.204681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:58.204972 (+ 291us) server_negotiation.cc:407] Connection header received
0928 23:47:58.205059 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:58.205062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:58.205128 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:58.205216 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:58.205521 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:58.206353 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:58.207508 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:47:58.207845 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:47:58.207959 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:47:58.208850 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:58.208854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:58.208857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:58.208870 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:58.208897 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:58.208922 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:58.209000 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:58.209330 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 23:47:58.209366 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:47:58.210440 9922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:58.215723 9922 catalog_manager.cc:1347] Generated new cluster ID: 799d2aeb5b004f3dad9f376f582018a9
I20260928 23:47:58.215796 9922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:58.241830 9922 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:58.243049 9922 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:58.251380 9922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2190d02c449f4fc08126768d1403efbe: Generated new TSK 0
I20260928 23:47:58.252133 9922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:47:58.261438 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9835
W20260928 23:47:58.267479 9820 connection.cc:535] client connection to 127.0.0.1:33891 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33891): Connection reset by peer (error 104)
2026-09-28T23:47:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:58 dist-test-slave-c3s8 krb5kdc[9934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:58 dist-test-slave-c3s8 krb5kdc[9934](info): set up 2 sockets
Sep 28 23:47:58 dist-test-slave-c3s8 krb5kdc[9934](info): commencing operation
krb5kdc: starting...
W20260928 23:47:59.314224 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:47:59 dist-test-slave-c3s8 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:47:59.444689 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:47:59.557701 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:47:59.557973 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:47:59.558017 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:47:59.562155 9947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:47:59.562211 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:47:59.562230 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:47:59.562247 9947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:47:59.562263 9947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:47:59.567401 9947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:47:59.568389 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:47:59.569341 9947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:47:59.578634 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:59.578658 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:47:59.578637 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:47:59.579011 9947 server_base.cc:1034] running on GCE node
I20260928 23:47:59.579571 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:47:59.580819 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:47:59.582038 9947 hybrid_clock.cc:648] HybridClock initialized: now 1790639279582007 us; error 28 us; skew 500 ppm
Sep 28 23:47:59 dist-test-slave-c3s8 krb5kdc[9934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:47:59.584857 9947 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:47:59.585877 9947 webserver.cc:460] Webserver started at http://127.0.0.1:38571/ using document root <none> and password file <none>
I20260928 23:47:59.586491 9947 fs_manager.cc:359] Metadata directory not provided
I20260928 23:47:59.586586 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:47:59.586823 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:47:59.588886 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "6fb46c72858f4476b33d3364c796945a"
format_stamp: "Formatted at 2026-09-28 23:47:59 on dist-test-slave-c3s8"
server_key: "6039d87bc0a6f0b44e11a64f2a5cd23d"
server_key_iv: "ea644dbade7fe5b8bc2f70ce83119903"
server_key_version: "encryptionkey@0"
I20260928 23:47:59.589493 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "6fb46c72858f4476b33d3364c796945a"
format_stamp: "Formatted at 2026-09-28 23:47:59 on dist-test-slave-c3s8"
server_key: "6039d87bc0a6f0b44e11a64f2a5cd23d"
server_key_iv: "ea644dbade7fe5b8bc2f70ce83119903"
server_key_version: "encryptionkey@0"
I20260928 23:47:59.593391 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 23:47:59.596009 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:47:59.597265 9947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:47:59.597447 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "6fb46c72858f4476b33d3364c796945a"
format_stamp: "Formatted at 2026-09-28 23:47:59 on dist-test-slave-c3s8"
server_key: "6039d87bc0a6f0b44e11a64f2a5cd23d"
server_key_iv: "ea644dbade7fe5b8bc2f70ce83119903"
server_key_version: "encryptionkey@0"
I20260928 23:47:59.597592 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:47:59.602283 9947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:47:59.605594 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:47:59.605798 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:47:59.613029 9947 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43111
I20260928 23:47:59.613054 10014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43111 every 8 connection(s)
I20260928 23:47:59.614053 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:47:59.616885 10015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:47:59.621289 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 9947
I20260928 23:47:59.621455 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:47:59.621783 21953 external_mini_cluster.cc:1428] Setting key 4a13f251ea8cda9e643b8c650076f817
I20260928 23:47:59.623250 10015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a: Bootstrap starting.
I20260928 23:47:59.625663 10015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a: Neither blocks nor log segments found. Creating new log.
I20260928 23:47:59.626451 10015 log.cc:824] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:47:59.627487 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:59.623405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53887 (local address 127.0.0.1:43111)
0928 23:47:59.623928 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:59.623943 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:59.623978 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:47:59.624684 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:59.624703 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:59.624995 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:59.625355 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:59.626140 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:59.626152 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:59.626164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:59.626174 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:59.626222 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:59.626249 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:59.626363 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:59.626472 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 23:47:59.626633 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":142,"threads_started":1}
I20260928 23:47:59.628512 10015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a: No bootstrap required, opened a new log
I20260928 23:47:59.630574 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:47:59.632880 10015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb46c72858f4476b33d3364c796945a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43111 } }
I20260928 23:47:59.633220 10015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:47:59.633288 10015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb46c72858f4476b33d3364c796945a, State: Initialized, Role: FOLLOWER
I20260928 23:47:59.633869 10015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb46c72858f4476b33d3364c796945a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43111 } }
I20260928 23:47:59.634066 10015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:47:59.634155 10015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:47:59.634290 10015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:47:59.634760 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:47:59.632774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33373 (local address 127.0.0.1:43111)
0928 23:47:59.632920 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:47:59.632924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:47:59.633136 (+ 212us) server_negotiation.cc:407] Connection header received
0928 23:47:59.633299 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:47:59.633302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:47:59.633343 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:47:59.633438 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:47:59.634368 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:47:59.634373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:47:59.634376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:47:59.634390 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:47:59.634423 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:47:59.634437 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:47:59.634534 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:47:59.634603 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:47:59.634646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:47:59.635579 10015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb46c72858f4476b33d3364c796945a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43111 } }
I20260928 23:47:59.635942 10015 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb46c72858f4476b33d3364c796945a; no voters:
I20260928 23:47:59.636265 10015 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:47:59.636411 10025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:47:59.636678 10025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [term 1 LEADER]: Becoming Leader. State: Replica: 6fb46c72858f4476b33d3364c796945a, State: Running, Role: LEADER
I20260928 23:47:59.637012 10025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb46c72858f4476b33d3364c796945a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43111 } }
I20260928 23:47:59.637713 10015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:47:59.639072 10026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb46c72858f4476b33d3364c796945a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb46c72858f4476b33d3364c796945a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43111 } } }
I20260928 23:47:59.639596 10026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [sys.catalog]: This master's current role is: LEADER
I20260928 23:47:59.639083 10027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb46c72858f4476b33d3364c796945a. Latest consensus state: current_term: 1 leader_uuid: "6fb46c72858f4476b33d3364c796945a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb46c72858f4476b33d3364c796945a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43111 } } }
I20260928 23:47:59.639892 10027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a [sys.catalog]: This master's current role is: LEADER
W20260928 23:47:59.642722 10040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:47:59.642813 10040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:47:59.642901 10039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:47:59.643561 10039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:47:59.649712 10039 catalog_manager.cc:1347] Generated new cluster ID: d3cda93c3eb94e8ca72c7f33d03cf72b
I20260928 23:47:59.649799 10039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:47:59.661149 10039 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:47:59.661504 10039 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:47:59.670950 10039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb46c72858f4476b33d3364c796945a: Generated new TSK 0
I20260928 23:47:59.671864 10039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:47:59.685497 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 9947
2026-09-28T23:47:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:47:59 dist-test-slave-c3s8 krb5kdc[10046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:47:59 dist-test-slave-c3s8 krb5kdc[10046](info): set up 2 sockets
Sep 28 23:47:59 dist-test-slave-c3s8 krb5kdc[10046](info): commencing operation
krb5kdc: starting...
W20260928 23:48:00.758313 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:00 dist-test-slave-c3s8 krb5kdc[10046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:00.886615 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:00.999768 10059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:01.000062 10059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:01.000106 10059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:01.004427 10059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:01.004493 10059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:01.004513 10059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:01.004530 10059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:01.004570 10059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:01.010238 10059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:01.011449 10059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:01.012534 10059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:01.021495 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:01.021476 10065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:01.021862 10059 server_base.cc:1034] running on GCE node
W20260928 23:48:01.021430 10064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:01.022606 10059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:01.023800 10059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:01.025027 10059 hybrid_clock.cc:648] HybridClock initialized: now 1790639281025003 us; error 41 us; skew 500 ppm
Sep 28 23:48:01 dist-test-slave-c3s8 krb5kdc[10046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:01.028277 10059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:01.029706 10059 webserver.cc:460] Webserver started at http://127.0.0.1:39539/ using document root <none> and password file <none>
I20260928 23:48:01.030368 10059 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:01.030464 10059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:01.030730 10059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:01.033044 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "52dd5a82dfdd4d9f965143d531788587"
format_stamp: "Formatted at 2026-09-28 23:48:01 on dist-test-slave-c3s8"
server_key: "72f6cb278432a4296921885cb80b40c8"
server_key_iv: "53f3ee245ab19609270c3895fba0742c"
server_key_version: "encryptionkey@0"
I20260928 23:48:01.033768 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "52dd5a82dfdd4d9f965143d531788587"
format_stamp: "Formatted at 2026-09-28 23:48:01 on dist-test-slave-c3s8"
server_key: "72f6cb278432a4296921885cb80b40c8"
server_key_iv: "53f3ee245ab19609270c3895fba0742c"
server_key_version: "encryptionkey@0"
I20260928 23:48:01.037951 10059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:48:01.040429 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:01.041867 10059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:01.042004 10059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "52dd5a82dfdd4d9f965143d531788587"
format_stamp: "Formatted at 2026-09-28 23:48:01 on dist-test-slave-c3s8"
server_key: "72f6cb278432a4296921885cb80b40c8"
server_key_iv: "53f3ee245ab19609270c3895fba0742c"
server_key_version: "encryptionkey@0"
I20260928 23:48:01.042120 10059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:01.055935 10059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:01.059217 10059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:01.059440 10059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:01.066639 10059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39299
I20260928 23:48:01.066660 10126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39299 every 8 connection(s)
I20260928 23:48:01.067649 10059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:01.070438 10127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:01.073674 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10059
I20260928 23:48:01.073836 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:01.074167 21953 external_mini_cluster.cc:1428] Setting key 58dce10dae188e03430ba27692216ae2
I20260928 23:48:01.077447 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587: Bootstrap starting.
I20260928 23:48:01.079727 10127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:01.080483 10127 log.cc:824] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:01.082557 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587: No bootstrap required, opened a new log
Sep 28 23:48:01 dist-test-slave-c3s8 krb5kdc[10046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:48:01.086417 10127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd5a82dfdd4d9f965143d531788587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260928 23:48:01.086668 10127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:01.086755 10127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dd5a82dfdd4d9f965143d531788587, State: Initialized, Role: FOLLOWER
I20260928 23:48:01.087252 10127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd5a82dfdd4d9f965143d531788587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260928 23:48:01.087379 10127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:01.087466 10127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:01.087581 10127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:01.088608 10127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd5a82dfdd4d9f965143d531788587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260928 23:48:01.088960 10127 leader_election.cc:302] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52dd5a82dfdd4d9f965143d531788587; no voters:
I20260928 23:48:01.089264 10127 leader_election.cc:288] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:01.089387 10132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:01.089432 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:01.075806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50633 (local address 127.0.0.1:39299)
0928 23:48:01.076285 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:01.076303 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:01.076341 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:48:01.076997 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:01.077046 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:01.077466 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:01.077881 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:01.078713 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:01.079740 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:01.080624 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:01.081130 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:01.081301 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:01.084137 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:01.084142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:01.084156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:01.084183 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:01.086053 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:01.086519 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:01.086524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:01.086530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:01.086644 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:01.087029 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:01.087033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:01.087035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:01.087615 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:01.087852 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:01.088143 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 23:48:01.088464 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":155,"threads_started":1}
I20260928 23:48:01.089645 10132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [term 1 LEADER]: Becoming Leader. State: Replica: 52dd5a82dfdd4d9f965143d531788587, State: Running, Role: LEADER
I20260928 23:48:01.089983 10132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd5a82dfdd4d9f965143d531788587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260928 23:48:01.090703 10127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:01.091524 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52dd5a82dfdd4d9f965143d531788587. Latest consensus state: current_term: 1 leader_uuid: "52dd5a82dfdd4d9f965143d531788587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd5a82dfdd4d9f965143d531788587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } } }
I20260928 23:48:01.091663 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:01.092489 10141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:01.092811 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:01.093106 10133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52dd5a82dfdd4d9f965143d531788587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd5a82dfdd4d9f965143d531788587" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } } }
I20260928 23:48:01.093238 10133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:01.095991 10141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:01.102241 10141 catalog_manager.cc:1347] Generated new cluster ID: 22947ef694d24395862f6d4737fbffbb
I20260928 23:48:01.102319 10141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:01.106281 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:01.095085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40891 (local address 127.0.0.1:39299)
0928 23:48:01.095200 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:01.095206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:01.098135 (+ 2929us) server_negotiation.cc:407] Connection header received
0928 23:48:01.098302 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:01.098305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:01.098358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:01.098446 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:01.099142 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:01.099702 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:01.100775 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:01.101203 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:01.101318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:01.102931 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:01.102934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:01.102937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:01.102962 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:01.104497 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:01.104951 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:01.104955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:01.104957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:01.105043 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:01.105337 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:01.105341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:01.105343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:01.105588 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:01.105732 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:01.105987 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 23:48:01.106104 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:48:01.110837 10141 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:01.112141 10141 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:01.128723 10141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52dd5a82dfdd4d9f965143d531788587: Generated new TSK 0
I20260928 23:48:01.129693 10141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:01.142434 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10059
2026-09-28T23:48:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:01 dist-test-slave-c3s8 krb5kdc[10158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:01 dist-test-slave-c3s8 krb5kdc[10158](info): set up 2 sockets
Sep 28 23:48:01 dist-test-slave-c3s8 krb5kdc[10158](info): commencing operation
krb5kdc: starting...
W20260928 23:48:02.194094 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:02 dist-test-slave-c3s8 krb5kdc[10158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:02.323146 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:02.437083 10171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:02.437379 10171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:02.437424 10171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:02.441586 10171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:02.441644 10171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:02.441664 10171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:02.441681 10171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:02.441696 10171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:02.447116 10171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:02.448163 10171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:02.449267 10171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:02.457981 10177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:02.458024 10179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:02.457994 10176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:02.458168 10171 server_base.cc:1034] running on GCE node
I20260928 23:48:02.458933 10171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:02.460268 10171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:02.461460 10171 hybrid_clock.cc:648] HybridClock initialized: now 1790639282461423 us; error 50 us; skew 500 ppm
Sep 28 23:48:02 dist-test-slave-c3s8 krb5kdc[10158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:02.464651 10171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:02.465744 10171 webserver.cc:460] Webserver started at http://127.0.0.1:35293/ using document root <none> and password file <none>
I20260928 23:48:02.466346 10171 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:02.466411 10171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:02.466593 10171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:02.468788 10171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "87f50b8c5fbe48e9ade0d4e705f31d82"
format_stamp: "Formatted at 2026-09-28 23:48:02 on dist-test-slave-c3s8"
server_key: "f7b9777a715531d980e7fbe5f03564b0"
server_key_iv: "9120b335d2a77563414930ed906dd137"
server_key_version: "encryptionkey@0"
I20260928 23:48:02.469386 10171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "87f50b8c5fbe48e9ade0d4e705f31d82"
format_stamp: "Formatted at 2026-09-28 23:48:02 on dist-test-slave-c3s8"
server_key: "f7b9777a715531d980e7fbe5f03564b0"
server_key_iv: "9120b335d2a77563414930ed906dd137"
server_key_version: "encryptionkey@0"
I20260928 23:48:02.473672 10171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:48:02.476389 10186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:02.478010 10171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:02.478157 10171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "87f50b8c5fbe48e9ade0d4e705f31d82"
format_stamp: "Formatted at 2026-09-28 23:48:02 on dist-test-slave-c3s8"
server_key: "f7b9777a715531d980e7fbe5f03564b0"
server_key_iv: "9120b335d2a77563414930ed906dd137"
server_key_version: "encryptionkey@0"
I20260928 23:48:02.478276 10171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:02.493279 10171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:02.496450 10171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:02.496667 10171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:02.504247 10171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42265
I20260928 23:48:02.504643 10238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42265 every 8 connection(s)
I20260928 23:48:02.505334 10171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:02.508541 10239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:02.510197 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10171
I20260928 23:48:02.510360 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:02.510682 21953 external_mini_cluster.cc:1428] Setting key dd935d505b7f1bf3aacdd1cfda1f4e9a
I20260928 23:48:02.515794 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82: Bootstrap starting.
I20260928 23:48:02.518496 10239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:02.519375 10239 log.cc:824] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:02.520622 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:02.512544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46185 (local address 127.0.0.1:42265)
0928 23:48:02.513063 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:02.513079 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:02.513118 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:48:02.513901 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:02.513922 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:02.514341 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:02.514730 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:02.515165 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:02.516218 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:02.517177 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:02.517758 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:02.517963 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:02.518927 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:02.518932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:02.518946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:02.519339 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:02.519394 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:02.519416 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:02.519526 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:02.519657 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:48:02.519876 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":154,"threads_started":1}
I20260928 23:48:02.521668 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82: No bootstrap required, opened a new log
I20260928 23:48:02.523756 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:02.526535 10239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260928 23:48:02.526888 10239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:02.526988 10239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f50b8c5fbe48e9ade0d4e705f31d82, State: Initialized, Role: FOLLOWER
I20260928 23:48:02.527524 10239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260928 23:48:02.527730 10239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:02.527798 10239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:02.527932 10239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:02.529242 10239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260928 23:48:02.529735 10239 leader_election.cc:302] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f50b8c5fbe48e9ade0d4e705f31d82; no voters:
I20260928 23:48:02.530313 10239 leader_election.cc:288] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:02.530401 10249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:02.530692 10249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [term 1 LEADER]: Becoming Leader. State: Replica: 87f50b8c5fbe48e9ade0d4e705f31d82, State: Running, Role: LEADER
I20260928 23:48:02.531076 10249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } }
I20260928 23:48:02.531217 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:02.525892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39301 (local address 127.0.0.1:42265)
0928 23:48:02.526029 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:02.526034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:02.526264 (+ 230us) server_negotiation.cc:407] Connection header received
0928 23:48:02.526381 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:02.526384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:02.526426 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:02.526507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:02.526878 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:02.527488 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:02.528727 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:02.528990 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:02.529076 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:02.530257 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:02.530262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:02.530264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:02.530279 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:02.530335 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:02.530357 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:02.530454 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:02.530715 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:48:02.530801 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:48:02.532011 10239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:02.532536 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } } }
I20260928 23:48:02.532655 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:02.532868 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f50b8c5fbe48e9ade0d4e705f31d82. Latest consensus state: current_term: 1 leader_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f50b8c5fbe48e9ade0d4e705f31d82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42265 } } }
I20260928 23:48:02.532968 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:02.533267 10253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:02.536916 10253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:02.542675 10253 catalog_manager.cc:1347] Generated new cluster ID: 554e4d59eb124f6cb9f69544488cbcab
I20260928 23:48:02.542757 10253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:02.563086 10253 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:02.564538 10253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:02.573419 10253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f50b8c5fbe48e9ade0d4e705f31d82: Generated new TSK 0
I20260928 23:48:02.574445 10253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:02.590097 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10171
2026-09-28T23:48:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:02 dist-test-slave-c3s8 krb5kdc[10270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:02 dist-test-slave-c3s8 krb5kdc[10270](info): set up 2 sockets
Sep 28 23:48:02 dist-test-slave-c3s8 krb5kdc[10270](info): commencing operation
krb5kdc: starting...
W20260928 23:48:03.666081 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:03 dist-test-slave-c3s8 krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:03.793758 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:03.905682 10283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:03.906018 10283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:03.906100 10283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:03.910444 10283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:03.910526 10283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:03.910561 10283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:03.910616 10283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:03.910652 10283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:03.915984 10283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:03.917112 10283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:03.918210 10283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:03.926890 10288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:03.926914 10291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:03.926981 10289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:03.927026 10283 server_base.cc:1034] running on GCE node
I20260928 23:48:03.927786 10283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:03.929333 10283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:03.930539 10283 hybrid_clock.cc:648] HybridClock initialized: now 1790639283930524 us; error 37 us; skew 500 ppm
Sep 28 23:48:03 dist-test-slave-c3s8 krb5kdc[10270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:03.933202 10283 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:03.934186 10283 webserver.cc:460] Webserver started at http://127.0.0.1:43519/ using document root <none> and password file <none>
I20260928 23:48:03.934721 10283 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:03.934778 10283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:03.934942 10283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:03.937002 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "64b8df78010b4d099c69e276facbf4d9"
format_stamp: "Formatted at 2026-09-28 23:48:03 on dist-test-slave-c3s8"
server_key: "2bd3af1d7f9a8b55bac0fec00d34e80c"
server_key_iv: "9ad9d8bc9bec94c9fd50eb913b78e414"
server_key_version: "encryptionkey@0"
I20260928 23:48:03.937525 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "64b8df78010b4d099c69e276facbf4d9"
format_stamp: "Formatted at 2026-09-28 23:48:03 on dist-test-slave-c3s8"
server_key: "2bd3af1d7f9a8b55bac0fec00d34e80c"
server_key_iv: "9ad9d8bc9bec94c9fd50eb913b78e414"
server_key_version: "encryptionkey@0"
I20260928 23:48:03.941239 10283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 23:48:03.943789 10298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:03.945089 10283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:48:03.945225 10283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "64b8df78010b4d099c69e276facbf4d9"
format_stamp: "Formatted at 2026-09-28 23:48:03 on dist-test-slave-c3s8"
server_key: "2bd3af1d7f9a8b55bac0fec00d34e80c"
server_key_iv: "9ad9d8bc9bec94c9fd50eb913b78e414"
server_key_version: "encryptionkey@0"
I20260928 23:48:03.945336 10283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:03.950103 10283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:03.953775 10283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:03.953976 10283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:03.961369 10283 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45229
I20260928 23:48:03.961418 10350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45229 every 8 connection(s)
I20260928 23:48:03.962448 10283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:03.965368 10351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:03.970466 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10283
I20260928 23:48:03.970633 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:03.970901 21953 external_mini_cluster.cc:1428] Setting key 01f9853755b0a17f90ead4ea271ec226
I20260928 23:48:03.971302 10351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9: Bootstrap starting.
I20260928 23:48:03.973892 10351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:03.974803 10351 log.cc:824] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:03.976137 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:03.972285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58509 (local address 127.0.0.1:45229)
0928 23:48:03.972711 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:03.972726 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:03.972763 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:48:03.973386 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:03.973402 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:03.973769 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:03.974109 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:03.974944 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:03.974952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:03.974970 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:03.974982 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:03.975042 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:03.975063 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:03.975179 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:03.975283 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 23:48:03.975446 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
I20260928 23:48:03.976855 10351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9: No bootstrap required, opened a new log
I20260928 23:48:03.978900 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:03.980939 10351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b8df78010b4d099c69e276facbf4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45229 } }
I20260928 23:48:03.981254 10351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:03.981362 10351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b8df78010b4d099c69e276facbf4d9, State: Initialized, Role: FOLLOWER
I20260928 23:48:03.981918 10351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b8df78010b4d099c69e276facbf4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45229 } }
I20260928 23:48:03.982106 10351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:03.982179 10351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:03.982312 10351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:03.982949 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:03.981114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57355 (local address 127.0.0.1:45229)
0928 23:48:03.981229 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:03.981234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:03.981403 (+ 169us) server_negotiation.cc:407] Connection header received
0928 23:48:03.981582 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:03.981585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:03.981629 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:03.981715 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:03.982548 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:03.982554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:03.982560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:03.982576 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:03.982609 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:03.982623 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:03.982718 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:03.982778 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 23:48:03.982824 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:48:03.983536 10351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b8df78010b4d099c69e276facbf4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45229 } }
I20260928 23:48:03.983911 10351 leader_election.cc:302] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64b8df78010b4d099c69e276facbf4d9; no voters:
I20260928 23:48:03.984272 10351 leader_election.cc:288] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:03.984344 10361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:03.984531 10361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [term 1 LEADER]: Becoming Leader. State: Replica: 64b8df78010b4d099c69e276facbf4d9, State: Running, Role: LEADER
I20260928 23:48:03.984862 10361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b8df78010b4d099c69e276facbf4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45229 } }
I20260928 23:48:03.985709 10351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:03.986845 10362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64b8df78010b4d099c69e276facbf4d9. Latest consensus state: current_term: 1 leader_uuid: "64b8df78010b4d099c69e276facbf4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b8df78010b4d099c69e276facbf4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45229 } } }
I20260928 23:48:03.986882 10363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64b8df78010b4d099c69e276facbf4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b8df78010b4d099c69e276facbf4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45229 } } }
I20260928 23:48:03.986989 10362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:03.987062 10363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:03.987457 10370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:03.990638 10370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:03.996876 10370 catalog_manager.cc:1347] Generated new cluster ID: 3e671a9c000441a5b8cc08b4c5f605e6
I20260928 23:48:03.996973 10370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:04.007946 10370 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:04.008315 10370 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:04.014526 10370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64b8df78010b4d099c69e276facbf4d9: Generated new TSK 0
I20260928 23:48:04.015424 10370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:04.022807 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10283
2026-09-28T23:48:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:04 dist-test-slave-c3s8 krb5kdc[10382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:04 dist-test-slave-c3s8 krb5kdc[10382](info): set up 2 sockets
Sep 28 23:48:04 dist-test-slave-c3s8 krb5kdc[10382](info): commencing operation
krb5kdc: starting...
W20260928 23:48:05.082252 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:05 dist-test-slave-c3s8 krb5kdc[10382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:05.216807 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:05.329524 10395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:05.329888 10395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:05.329936 10395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:05.334122 10395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:05.334188 10395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:05.334208 10395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:05.334225 10395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:05.334301 10395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:05.339818 10395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:05.341037 10395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:05.342099 10395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:05.350766 10400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:05.350893 10401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:05.350893 10403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:05.350973 10395 server_base.cc:1034] running on GCE node
I20260928 23:48:05.351588 10395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:05.352720 10395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:05.353929 10395 hybrid_clock.cc:648] HybridClock initialized: now 1790639285353898 us; error 46 us; skew 500 ppm
Sep 28 23:48:05 dist-test-slave-c3s8 krb5kdc[10382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:05.362922 10395 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:05.363878 10395 webserver.cc:460] Webserver started at http://127.0.0.1:44357/ using document root <none> and password file <none>
I20260928 23:48:05.364398 10395 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:05.364455 10395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:05.364617 10395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:05.366750 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "278f1093dfa54074b28850c7ad3dbc2b"
format_stamp: "Formatted at 2026-09-28 23:48:05 on dist-test-slave-c3s8"
server_key: "bd194a882050bc03a1f7cea5be74f9d4"
server_key_iv: "c26dc548af990804f5c97eb1c0cc149b"
server_key_version: "encryptionkey@0"
I20260928 23:48:05.367256 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "278f1093dfa54074b28850c7ad3dbc2b"
format_stamp: "Formatted at 2026-09-28 23:48:05 on dist-test-slave-c3s8"
server_key: "bd194a882050bc03a1f7cea5be74f9d4"
server_key_iv: "c26dc548af990804f5c97eb1c0cc149b"
server_key_version: "encryptionkey@0"
I20260928 23:48:05.371038 10395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:48:05.373742 10410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:05.375188 10395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:05.375339 10395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "278f1093dfa54074b28850c7ad3dbc2b"
format_stamp: "Formatted at 2026-09-28 23:48:05 on dist-test-slave-c3s8"
server_key: "bd194a882050bc03a1f7cea5be74f9d4"
server_key_iv: "c26dc548af990804f5c97eb1c0cc149b"
server_key_version: "encryptionkey@0"
I20260928 23:48:05.375461 10395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:05.387509 10395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:05.390987 10395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:05.391218 10395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:05.398581 10395 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34871
I20260928 23:48:05.398605 10462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34871 every 8 connection(s)
I20260928 23:48:05.399544 10395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:05.402387 10463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:05.403764 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10395
I20260928 23:48:05.403905 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:05.404206 21953 external_mini_cluster.cc:1428] Setting key 973360a20a7a96298bdde48f945ed3fe
I20260928 23:48:05.408377 10463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b: Bootstrap starting.
I20260928 23:48:05.410648 10463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:05.411387 10463 log.cc:824] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:48:05 dist-test-slave-c3s8 krb5kdc[10382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:48:05.413615 10463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b: No bootstrap required, opened a new log
I20260928 23:48:05.418504 10463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278f1093dfa54074b28850c7ad3dbc2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260928 23:48:05.418803 10463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:05.418860 10463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278f1093dfa54074b28850c7ad3dbc2b, State: Initialized, Role: FOLLOWER
I20260928 23:48:05.419342 10463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278f1093dfa54074b28850c7ad3dbc2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260928 23:48:05.419498 10463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:05.419569 10463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:05.419622 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:05.405716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:58905 (local address 127.0.0.1:34871)
0928 23:48:05.406169 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:05.406184 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:05.406222 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:48:05.406887 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:05.406906 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:05.407250 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:05.407596 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:05.408426 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:05.409441 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:05.410387 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:05.410936 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:05.411111 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:05.414065 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:05.414071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:05.414087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:05.414117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:05.416212 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:05.416789 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:05.416795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:05.416801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:05.416940 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:05.417290 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:05.417293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:05.417295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:05.417848 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:05.418060 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:05.418387 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 23:48:05.418622 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":147,"threads_started":1}
I20260928 23:48:05.419672 10463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:05.420656 10463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278f1093dfa54074b28850c7ad3dbc2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260928 23:48:05.421012 10463 leader_election.cc:302] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278f1093dfa54074b28850c7ad3dbc2b; no voters:
I20260928 23:48:05.421350 10463 leader_election.cc:288] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:05.421427 10468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:05.421674 10468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [term 1 LEADER]: Becoming Leader. State: Replica: 278f1093dfa54074b28850c7ad3dbc2b, State: Running, Role: LEADER
I20260928 23:48:05.421975 10468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278f1093dfa54074b28850c7ad3dbc2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260928 23:48:05.422691 10463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:05.422762 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:05.423137 10468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278f1093dfa54074b28850c7ad3dbc2b. Latest consensus state: current_term: 1 leader_uuid: "278f1093dfa54074b28850c7ad3dbc2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278f1093dfa54074b28850c7ad3dbc2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } } }
I20260928 23:48:05.423251 10468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:05.423467 10468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278f1093dfa54074b28850c7ad3dbc2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278f1093dfa54074b28850c7ad3dbc2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } } }
I20260928 23:48:05.423571 10468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b [sys.catalog]: This master's current role is: LEADER
W20260928 23:48:05.428042 10487 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:05.428140 10487 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:05.428246 10472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:05.429124 10472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:05.434320 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:05.425134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:32957 (local address 127.0.0.1:34871)
0928 23:48:05.425245 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:05.425249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:05.425616 (+ 367us) server_negotiation.cc:407] Connection header received
0928 23:48:05.425742 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:05.425744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:05.425792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:05.425879 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:05.426504 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:05.427238 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:05.428488 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:05.428822 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:05.428933 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:05.430771 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:05.430774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:05.430778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:05.430804 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:05.432340 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:05.432840 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:05.432844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:05.432846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:05.432926 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:05.433299 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:05.433302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:05.433303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:05.433519 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:05.433679 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:05.434042 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 23:48:05.434138 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:48:05.435736 10472 catalog_manager.cc:1347] Generated new cluster ID: 61a0295ca0e9448c946ca9b435602889
I20260928 23:48:05.435812 10472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:05.453642 10472 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:05.455019 10472 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:05.462317 10472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278f1093dfa54074b28850c7ad3dbc2b: Generated new TSK 0
I20260928 23:48:05.463174 10472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:05.468571 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10395
W20260928 23:48:05.474084 10380 connection.cc:535] client connection to 127.0.0.1:34871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34871): Connection reset by peer (error 104)
2026-09-28T23:48:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:05 dist-test-slave-c3s8 krb5kdc[10494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:05 dist-test-slave-c3s8 krb5kdc[10494](info): set up 2 sockets
Sep 28 23:48:05 dist-test-slave-c3s8 krb5kdc[10494](info): commencing operation
krb5kdc: starting...
W20260928 23:48:06.542269 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:06 dist-test-slave-c3s8 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:06.669332 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:06.781627 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:06.781924 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:06.781967 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:06.786147 10507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:06.786213 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:06.786233 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:06.786253 10507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:06.786268 10507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:06.791673 10507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:06.792824 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:06.794011 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:48:06.802980 10507 server_base.cc:1034] running on GCE node
W20260928 23:48:06.802922 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:06.802896 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:06.802922 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:06.803853 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:06.805022 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:06.806200 10507 hybrid_clock.cc:648] HybridClock initialized: now 1790639286806182 us; error 30 us; skew 500 ppm
Sep 28 23:48:06 dist-test-slave-c3s8 krb5kdc[10494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:06.808974 10507 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:06.810031 10507 webserver.cc:460] Webserver started at http://127.0.0.1:36603/ using document root <none> and password file <none>
I20260928 23:48:06.810611 10507 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:06.810703 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:06.810940 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:06.813015 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "74f294192aed4e4a851dda86dabb6897"
format_stamp: "Formatted at 2026-09-28 23:48:06 on dist-test-slave-c3s8"
server_key: "8c80532aff3f1b1f46238adab8d52b67"
server_key_iv: "00449ae3801efd4ba588630a3bb541d8"
server_key_version: "encryptionkey@0"
I20260928 23:48:06.813678 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "74f294192aed4e4a851dda86dabb6897"
format_stamp: "Formatted at 2026-09-28 23:48:06 on dist-test-slave-c3s8"
server_key: "8c80532aff3f1b1f46238adab8d52b67"
server_key_iv: "00449ae3801efd4ba588630a3bb541d8"
server_key_version: "encryptionkey@0"
I20260928 23:48:06.817392 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:06.819896 10522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:06.821336 10507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:48:06.821506 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "74f294192aed4e4a851dda86dabb6897"
format_stamp: "Formatted at 2026-09-28 23:48:06 on dist-test-slave-c3s8"
server_key: "8c80532aff3f1b1f46238adab8d52b67"
server_key_iv: "00449ae3801efd4ba588630a3bb541d8"
server_key_version: "encryptionkey@0"
I20260928 23:48:06.821698 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:06.855285 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:06.858629 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:06.858850 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:06.866330 10507 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36289
I20260928 23:48:06.866357 10574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36289 every 8 connection(s)
I20260928 23:48:06.867280 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:06.870121 10575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:06.875874 10575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897: Bootstrap starting.
I20260928 23:48:06.876295 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10507
I20260928 23:48:06.876431 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:06.876737 21953 external_mini_cluster.cc:1428] Setting key a6aa7900d51531356c09a0f092ff014d
I20260928 23:48:06.878793 10575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:06.879530 10575 log.cc:824] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:06.881846 10575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897: No bootstrap required, opened a new log
I20260928 23:48:06.885481 10575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f294192aed4e4a851dda86dabb6897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260928 23:48:06.885787 10575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:06.885874 10575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f294192aed4e4a851dda86dabb6897, State: Initialized, Role: FOLLOWER
I20260928 23:48:06.886346 10575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f294192aed4e4a851dda86dabb6897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260928 23:48:06.886488 10575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:06.886492 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:06.878224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:58897 (local address 127.0.0.1:36289)
0928 23:48:06.878665 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:06.878681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:06.878718 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:48:06.879502 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:06.879521 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:06.879976 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:06.880423 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:06.880853 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:06.881912 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:06.882848 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:06.883335 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:06.883516 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:06.884570 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:06.884575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:06.884588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:06.885005 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:06.885061 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:06.885085 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:06.885230 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:06.885418 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:48:06.885613 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260928 23:48:06.886584 10575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:06.886756 10575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:06.887661 10575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f294192aed4e4a851dda86dabb6897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260928 23:48:06.888036 10575 leader_election.cc:302] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f294192aed4e4a851dda86dabb6897; no voters:
I20260928 23:48:06.888401 10575 leader_election.cc:288] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:06.888445 10580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:06.888963 10580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [term 1 LEADER]: Becoming Leader. State: Replica: 74f294192aed4e4a851dda86dabb6897, State: Running, Role: LEADER
I20260928 23:48:06.889258 10580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f294192aed4e4a851dda86dabb6897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } }
I20260928 23:48:06.889572 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:06.889779 10575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:06.891033 10582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f294192aed4e4a851dda86dabb6897. Latest consensus state: current_term: 1 leader_uuid: "74f294192aed4e4a851dda86dabb6897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f294192aed4e4a851dda86dabb6897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } } }
I20260928 23:48:06.891139 10582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:06.891445 10581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f294192aed4e4a851dda86dabb6897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f294192aed4e4a851dda86dabb6897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36289 } } }
I20260928 23:48:06.891553 10581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:06.892472 10593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:06.895586 10593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:06.898799 10578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:06.892308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:43759 (local address 127.0.0.1:36289)
0928 23:48:06.892421 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:06.892424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:06.894097 (+ 1673us) server_negotiation.cc:407] Connection header received
0928 23:48:06.894246 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:06.894250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:06.894301 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:06.894391 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:06.894729 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:06.895456 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:06.896732 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:06.897081 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:06.897188 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:06.898250 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:06.898254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:06.898256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:06.898271 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:06.898301 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:06.898314 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:06.898417 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:06.898625 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 23:48:06.898662 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:48:06.901517 10593 catalog_manager.cc:1347] Generated new cluster ID: 0bf56373fb1e403ea8b5e696095e77b9
I20260928 23:48:06.901651 10593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:06.922900 10593 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:06.923974 10593 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:06.936854 10593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f294192aed4e4a851dda86dabb6897: Generated new TSK 0
I20260928 23:48:06.937837 10593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:06.944787 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10507
W20260928 23:48:06.950588 10492 connection.cc:535] client connection to 127.0.0.1:36289 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36289): Connection reset by peer (error 104)
2026-09-28T23:48:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:06 dist-test-slave-c3s8 krb5kdc[10606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:07 dist-test-slave-c3s8 krb5kdc[10606](info): set up 2 sockets
Sep 28 23:48:07 dist-test-slave-c3s8 krb5kdc[10606](info): commencing operation
krb5kdc: starting...
W20260928 23:48:08.006242 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:08 dist-test-slave-c3s8 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:08.133457 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:08.246012 10619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:08.246333 10619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:08.246417 10619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:08.250439 10619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:08.250545 10619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:08.250584 10619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:08.250618 10619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:08.250648 10619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:08.256014 10619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:08.257150 10619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:08.258240 10619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:08.267505 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:08.267717 10619 server_base.cc:1034] running on GCE node
W20260928 23:48:08.267505 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:08.267525 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:08.268559 10619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:08.269868 10619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:08.271075 10619 hybrid_clock.cc:648] HybridClock initialized: now 1790639288270980 us; error 110 us; skew 500 ppm
Sep 28 23:48:08 dist-test-slave-c3s8 krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:08.273907 10619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:08.274869 10619 webserver.cc:460] Webserver started at http://127.0.0.1:37401/ using document root <none> and password file <none>
I20260928 23:48:08.275460 10619 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:08.275561 10619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:08.275800 10619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:08.278007 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "e5c2d6612f6e4c649e751024d55da5b4"
format_stamp: "Formatted at 2026-09-28 23:48:08 on dist-test-slave-c3s8"
server_key: "19bc47221b1fc5b4ef1f7e293e80ed38"
server_key_iv: "571b358a817301fbeabfa1cca89054a9"
server_key_version: "encryptionkey@0"
I20260928 23:48:08.278589 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "e5c2d6612f6e4c649e751024d55da5b4"
format_stamp: "Formatted at 2026-09-28 23:48:08 on dist-test-slave-c3s8"
server_key: "19bc47221b1fc5b4ef1f7e293e80ed38"
server_key_iv: "571b358a817301fbeabfa1cca89054a9"
server_key_version: "encryptionkey@0"
I20260928 23:48:08.282601 10619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:48:08.285158 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:08.286497 10619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:48:08.286649 10619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "e5c2d6612f6e4c649e751024d55da5b4"
format_stamp: "Formatted at 2026-09-28 23:48:08 on dist-test-slave-c3s8"
server_key: "19bc47221b1fc5b4ef1f7e293e80ed38"
server_key_iv: "571b358a817301fbeabfa1cca89054a9"
server_key_version: "encryptionkey@0"
I20260928 23:48:08.286774 10619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:08.291705 10619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:08.294942 10619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:08.295171 10619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:08.302618 10619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33833
I20260928 23:48:08.302675 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33833 every 8 connection(s)
I20260928 23:48:08.303689 10619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:08.306632 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:08.310150 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10619
I20260928 23:48:08.310263 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:08.310662 21953 external_mini_cluster.cc:1428] Setting key 33966d083135ef9ec535540314aac712
I20260928 23:48:08.312705 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4: Bootstrap starting.
I20260928 23:48:08.315196 10687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:08.315973 10687 log.cc:824] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:08.316689 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:08.312063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:59743 (local address 127.0.0.1:33833)
0928 23:48:08.312550 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:08.312578 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:08.312899 (+ 321us) server_negotiation.cc:407] Connection header received
0928 23:48:08.313812 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:08.313831 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:08.314139 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:08.314499 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:08.315362 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:08.315366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:08.315379 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:08.315391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:08.315445 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:08.315467 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:08.315587 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:08.315690 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:48:08.315864 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":182,"threads_started":1}
I20260928 23:48:08.318037 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4: No bootstrap required, opened a new log
I20260928 23:48:08.319703 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:08.322516 10687 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2d6612f6e4c649e751024d55da5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33833 } }
I20260928 23:48:08.322846 10687 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:08.322916 10687 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c2d6612f6e4c649e751024d55da5b4, State: Initialized, Role: FOLLOWER
I20260928 23:48:08.323447 10687 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2d6612f6e4c649e751024d55da5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33833 } }
I20260928 23:48:08.323637 10687 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:08.323722 10687 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:08.323832 10687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:08.323944 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:08.322099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:39387 (local address 127.0.0.1:33833)
0928 23:48:08.322232 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:08.322236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:08.322491 (+ 255us) server_negotiation.cc:407] Connection header received
0928 23:48:08.322603 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:08.322605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:08.322636 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:08.322704 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:08.323428 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:08.323431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:08.323433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:08.323444 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:08.323470 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:08.323480 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:08.323585 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:08.323720 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:48:08.323762 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:48:08.324880 10687 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2d6612f6e4c649e751024d55da5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33833 } }
I20260928 23:48:08.325250 10687 leader_election.cc:302] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c2d6612f6e4c649e751024d55da5b4; no voters:
I20260928 23:48:08.325609 10687 leader_election.cc:288] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:08.325754 10697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:08.326026 10697 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [term 1 LEADER]: Becoming Leader. State: Replica: e5c2d6612f6e4c649e751024d55da5b4, State: Running, Role: LEADER
I20260928 23:48:08.326453 10697 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2d6612f6e4c649e751024d55da5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33833 } }
I20260928 23:48:08.326985 10687 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:08.328339 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c2d6612f6e4c649e751024d55da5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2d6612f6e4c649e751024d55da5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33833 } } }
I20260928 23:48:08.328481 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:08.328881 10699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c2d6612f6e4c649e751024d55da5b4. Latest consensus state: current_term: 1 leader_uuid: "e5c2d6612f6e4c649e751024d55da5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c2d6612f6e4c649e751024d55da5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33833 } } }
I20260928 23:48:08.328980 10699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4 [sys.catalog]: This master's current role is: LEADER
W20260928 23:48:08.332019 10712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:08.332103 10712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:08.332197 10707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:08.332917 10707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:08.339758 10707 catalog_manager.cc:1347] Generated new cluster ID: a8d48de624bc48eb813ec49bf446a43b
I20260928 23:48:08.339896 10707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:08.361101 10707 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:08.361465 10707 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:08.370447 10707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5c2d6612f6e4c649e751024d55da5b4: Generated new TSK 0
I20260928 23:48:08.371315 10707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:08.381774 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10619
2026-09-28T23:48:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:08 dist-test-slave-c3s8 krb5kdc[10718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:08 dist-test-slave-c3s8 krb5kdc[10718](info): set up 2 sockets
Sep 28 23:48:08 dist-test-slave-c3s8 krb5kdc[10718](info): commencing operation
krb5kdc: starting...
W20260928 23:48:09.462090 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:09 dist-test-slave-c3s8 krb5kdc[10718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:09.594370 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:09.706898 10731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:09.707252 10731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:09.707327 10731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:09.711462 10731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:09.711577 10731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:09.711618 10731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:09.711653 10731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:09.711684 10731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:09.717195 10731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:09.718451 10731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:09.719576 10731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:09.728672 10739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:09.728672 10736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:09.728690 10737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:09.729189 10731 server_base.cc:1034] running on GCE node
I20260928 23:48:09.729844 10731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:09.731094 10731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:09.732336 10731 hybrid_clock.cc:648] HybridClock initialized: now 1790639289732321 us; error 27 us; skew 500 ppm
Sep 28 23:48:09 dist-test-slave-c3s8 krb5kdc[10718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:09.735608 10731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:09.736699 10731 webserver.cc:460] Webserver started at http://127.0.0.1:45313/ using document root <none> and password file <none>
I20260928 23:48:09.737321 10731 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:09.737423 10731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:09.737722 10731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:09.739933 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "f21190921a6040809f05e95fed287467"
format_stamp: "Formatted at 2026-09-28 23:48:09 on dist-test-slave-c3s8"
server_key: "b2dbbf91184e1956eafb8e1a0019e196"
server_key_iv: "55903cc6b9da479d68125cd4983579dc"
server_key_version: "encryptionkey@0"
I20260928 23:48:09.740540 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "f21190921a6040809f05e95fed287467"
format_stamp: "Formatted at 2026-09-28 23:48:09 on dist-test-slave-c3s8"
server_key: "b2dbbf91184e1956eafb8e1a0019e196"
server_key_iv: "55903cc6b9da479d68125cd4983579dc"
server_key_version: "encryptionkey@0"
I20260928 23:48:09.744688 10731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:48:09.747493 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:09.749017 10731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:48:09.749198 10731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "f21190921a6040809f05e95fed287467"
format_stamp: "Formatted at 2026-09-28 23:48:09 on dist-test-slave-c3s8"
server_key: "b2dbbf91184e1956eafb8e1a0019e196"
server_key_iv: "55903cc6b9da479d68125cd4983579dc"
server_key_version: "encryptionkey@0"
I20260928 23:48:09.749333 10731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:09.769887 10731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:09.773473 10731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:09.773751 10731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:09.781275 10731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39199
I20260928 23:48:09.781291 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39199 every 8 connection(s)
I20260928 23:48:09.782315 10731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:09.785261 10799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:09.790985 10799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467: Bootstrap starting.
I20260928 23:48:09.791290 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10731
I20260928 23:48:09.791433 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:09.791724 21953 external_mini_cluster.cc:1428] Setting key 98f195bb3264337cc0d1a4302a33cbbc
I20260928 23:48:09.793893 10799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:09.794837 10799 log.cc:824] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:09.796864 10799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467: No bootstrap required, opened a new log
I20260928 23:48:09.800537 10799 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21190921a6040809f05e95fed287467" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260928 23:48:09.800800 10799 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:09.800882 10799 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21190921a6040809f05e95fed287467, State: Initialized, Role: FOLLOWER
Sep 28 23:48:09 dist-test-slave-c3s8 krb5kdc[10718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:48:09.801415 10799 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21190921a6040809f05e95fed287467" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260928 23:48:09.801651 10799 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:09.801734 10799 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:09.801862 10799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:09.803129 10799 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21190921a6040809f05e95fed287467" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260928 23:48:09.803483 10799 leader_election.cc:302] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21190921a6040809f05e95fed287467; no voters:
I20260928 23:48:09.803854 10799 leader_election.cc:288] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:09.803963 10804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:09.804188 10804 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [term 1 LEADER]: Becoming Leader. State: Replica: f21190921a6040809f05e95fed287467, State: Running, Role: LEADER
I20260928 23:48:09.804549 10804 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21190921a6040809f05e95fed287467" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260928 23:48:09.805357 10799 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:09.806278 10805 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21190921a6040809f05e95fed287467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21190921a6040809f05e95fed287467" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260928 23:48:09.806419 10805 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:09.806842 10806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21190921a6040809f05e95fed287467. Latest consensus state: current_term: 1 leader_uuid: "f21190921a6040809f05e95fed287467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21190921a6040809f05e95fed287467" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260928 23:48:09.806999 10806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:09.808245 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:09.793170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:50929 (local address 127.0.0.1:39199)
0928 23:48:09.793621 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:09.793642 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:09.793685 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:48:09.794606 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:09.794634 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:09.795111 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:09.795530 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:09.796353 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:09.797439 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:09.798351 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:09.798892 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:09.799080 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:09.801947 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:09.801953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:09.801971 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:09.802007 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:09.803938 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:09.804632 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:09.804637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:09.804643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:09.804784 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:09.805198 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:09.805201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:09.805203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:09.805750 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:09.805984 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:09.806993 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0928 23:48:09.807227 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
W20260928 23:48:09.810851 10820 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:09.810964 10820 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:09.811064 10813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:09.811101 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:09.811915 10813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:09.818349 10813 catalog_manager.cc:1347] Generated new cluster ID: de548d2ff8544591a27d2eaa9ef7df68
I20260928 23:48:09.818679 10813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:09.821203 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:09.813098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:35209 (local address 127.0.0.1:39199)
0928 23:48:09.813207 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:09.813212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:09.813438 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:48:09.813612 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:09.813615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:09.813658 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:09.813750 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:09.814411 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:09.814976 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:09.815799 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:09.816046 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:09.816161 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:09.817588 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:09.817592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:09.817594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:09.817619 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:09.819307 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:09.819876 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:09.819880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:09.819882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:09.819959 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:09.820278 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:09.820281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:09.820282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:09.820473 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:09.820577 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:09.820959 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 23:48:09.821054 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:48:09.857453 10813 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:09.858671 10813 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:09.866027 10813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21190921a6040809f05e95fed287467: Generated new TSK 0
I20260928 23:48:09.866793 10813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:09.876631 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10731
W20260928 23:48:09.882306 10716 connection.cc:535] client connection to 127.0.0.1:39199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39199): Connection reset by peer (error 104)
2026-09-28T23:48:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:09 dist-test-slave-c3s8 krb5kdc[10830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:09 dist-test-slave-c3s8 krb5kdc[10830](info): set up 2 sockets
Sep 28 23:48:09 dist-test-slave-c3s8 krb5kdc[10830](info): commencing operation
krb5kdc: starting...
W20260928 23:48:10.930269 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:11 dist-test-slave-c3s8 krb5kdc[10830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:11.065737 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:11.181178 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:11.181629 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:11.181777 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:11.185868 10843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:11.185978 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:11.186019 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:11.186056 10843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:11.186092 10843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:11.191787 10843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:11.192972 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:11.194087 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:11.203318 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:11.203276 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:11.203456 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:11.203570 10843 server_base.cc:1034] running on GCE node
I20260928 23:48:11.204169 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:11.205444 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:11.206676 10843 hybrid_clock.cc:648] HybridClock initialized: now 1790639291206654 us; error 60 us; skew 500 ppm
Sep 28 23:48:11 dist-test-slave-c3s8 krb5kdc[10830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:11.215969 10843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:11.217062 10843 webserver.cc:460] Webserver started at http://127.0.0.1:44031/ using document root <none> and password file <none>
I20260928 23:48:11.217756 10843 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:11.217849 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:11.218092 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:11.220237 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "74704d074c214c32afde96c2996b11d0"
format_stamp: "Formatted at 2026-09-28 23:48:11 on dist-test-slave-c3s8"
server_key: "02b8bfa37999259bb4a6251412a9dcfc"
server_key_iv: "de328a99d6e5eabb1aa0975faf1cfad2"
server_key_version: "encryptionkey@0"
I20260928 23:48:11.220817 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "74704d074c214c32afde96c2996b11d0"
format_stamp: "Formatted at 2026-09-28 23:48:11 on dist-test-slave-c3s8"
server_key: "02b8bfa37999259bb4a6251412a9dcfc"
server_key_iv: "de328a99d6e5eabb1aa0975faf1cfad2"
server_key_version: "encryptionkey@0"
I20260928 23:48:11.224782 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:48:11.227376 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:11.228688 10843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:48:11.228839 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "74704d074c214c32afde96c2996b11d0"
format_stamp: "Formatted at 2026-09-28 23:48:11 on dist-test-slave-c3s8"
server_key: "02b8bfa37999259bb4a6251412a9dcfc"
server_key_iv: "de328a99d6e5eabb1aa0975faf1cfad2"
server_key_version: "encryptionkey@0"
I20260928 23:48:11.228965 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:11.249274 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:11.252894 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:11.253177 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:11.260577 10843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32939
I20260928 23:48:11.260608 10910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32939 every 8 connection(s)
I20260928 23:48:11.261631 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:11.263055 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10843
I20260928 23:48:11.263196 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:11.263573 21953 external_mini_cluster.cc:1428] Setting key 2892958953b30fb19e8c0f3e3883f6d6
I20260928 23:48:11.265003 10911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:11.271085 10911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0: Bootstrap starting.
I20260928 23:48:11.273111 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:11.265083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:46163 (local address 127.0.0.1:32939)
0928 23:48:11.265778 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:11.265794 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:11.265830 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:48:11.266498 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:11.266516 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:11.266868 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:11.267219 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:11.267653 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:11.268634 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:11.269611 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:11.270154 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:11.270339 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:11.271367 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:11.271373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:11.271386 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:11.271786 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:11.271841 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:11.271868 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:11.272020 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:11.272213 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 23:48:11.272366 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":150,"threads_started":1}
I20260928 23:48:11.273666 10911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:11.274555 10911 log.cc:824] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:11.276484 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:11.276979 10911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0: No bootstrap required, opened a new log
I20260928 23:48:11.281471 10911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74704d074c214c32afde96c2996b11d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260928 23:48:11.281826 10911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:11.281919 10911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74704d074c214c32afde96c2996b11d0, State: Initialized, Role: FOLLOWER
I20260928 23:48:11.282416 10911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74704d074c214c32afde96c2996b11d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260928 23:48:11.282545 10911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:11.282629 10911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:11.282739 10911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:11.283216 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:11.278602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:40523 (local address 127.0.0.1:32939)
0928 23:48:11.278734 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:11.278739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:11.278886 (+ 147us) server_negotiation.cc:407] Connection header received
0928 23:48:11.278982 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:11.278985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:11.279041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:11.279143 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:11.279398 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:11.280190 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:11.281084 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:11.281440 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:11.281579 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:11.282721 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:11.282724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:11.282727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:11.282742 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:11.282774 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:11.282790 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:11.282890 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:11.283037 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 23:48:11.283090 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:48:11.283872 10911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74704d074c214c32afde96c2996b11d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260928 23:48:11.284238 10911 leader_election.cc:302] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74704d074c214c32afde96c2996b11d0; no voters:
I20260928 23:48:11.284677 10911 leader_election.cc:288] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:11.284777 10921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:11.285053 10921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [term 1 LEADER]: Becoming Leader. State: Replica: 74704d074c214c32afde96c2996b11d0, State: Running, Role: LEADER
I20260928 23:48:11.285429 10921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74704d074c214c32afde96c2996b11d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260928 23:48:11.286070 10911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:11.287262 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74704d074c214c32afde96c2996b11d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74704d074c214c32afde96c2996b11d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } } }
I20260928 23:48:11.287416 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:11.287801 10930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:11.287801 10923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74704d074c214c32afde96c2996b11d0. Latest consensus state: current_term: 1 leader_uuid: "74704d074c214c32afde96c2996b11d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74704d074c214c32afde96c2996b11d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } } }
I20260928 23:48:11.290500 10923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:11.291282 10930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:11.297382 10930 catalog_manager.cc:1347] Generated new cluster ID: 0d77aead7d57452380a70abb120f4bf8
I20260928 23:48:11.297501 10930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:11.315256 10930 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:11.316334 10930 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:11.329708 10930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74704d074c214c32afde96c2996b11d0: Generated new TSK 0
I20260928 23:48:11.330866 10930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:11.344405 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10843
W20260928 23:48:11.350116 10828 connection.cc:535] client connection to 127.0.0.1:32939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32939): Connection reset by peer (error 104)
2026-09-28T23:48:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:11 dist-test-slave-c3s8 krb5kdc[10942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:11 dist-test-slave-c3s8 krb5kdc[10942](info): set up 2 sockets
Sep 28 23:48:11 dist-test-slave-c3s8 krb5kdc[10942](info): commencing operation
krb5kdc: starting...
W20260928 23:48:12.394279 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:12 dist-test-slave-c3s8 krb5kdc[10942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:48:12.529058 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:12.646633 10955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:12.646970 10955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:12.647054 10955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:12.651213 10955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:12.651315 10955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:12.651355 10955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:12.651417 10955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:12.651458 10955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:12.657088 10955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:12.658275 10955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:12.659358 10955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:12.668723 10963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:12.668699 10960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:12.668797 10955 server_base.cc:1034] running on GCE node
W20260928 23:48:12.668718 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:12.669754 10955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:12.670930 10955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:12.672111 10955 hybrid_clock.cc:648] HybridClock initialized: now 1790639292672092 us; error 32 us; skew 500 ppm
Sep 28 23:48:12 dist-test-slave-c3s8 krb5kdc[10942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:12.675225 10955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:48:12.676227 10955 webserver.cc:460] Webserver started at http://127.0.0.1:44341/ using document root <none> and password file <none>
I20260928 23:48:12.676792 10955 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:12.676853 10955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:12.677031 10955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:12.679468 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4"
format_stamp: "Formatted at 2026-09-28 23:48:12 on dist-test-slave-c3s8"
server_key: "444072442e98130fe9beb748157dfb66"
server_key_iv: "77aa948ea6544eeb04ead6146ddaf3ab"
server_key_version: "encryptionkey@0"
I20260928 23:48:12.680047 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4"
format_stamp: "Formatted at 2026-09-28 23:48:12 on dist-test-slave-c3s8"
server_key: "444072442e98130fe9beb748157dfb66"
server_key_iv: "77aa948ea6544eeb04ead6146ddaf3ab"
server_key_version: "encryptionkey@0"
I20260928 23:48:12.684012 10955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:48:12.686604 10970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:12.687848 10955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:48:12.687979 10955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4"
format_stamp: "Formatted at 2026-09-28 23:48:12 on dist-test-slave-c3s8"
server_key: "444072442e98130fe9beb748157dfb66"
server_key_iv: "77aa948ea6544eeb04ead6146ddaf3ab"
server_key_version: "encryptionkey@0"
I20260928 23:48:12.688082 10955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:12.692704 10955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:12.695920 10955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:12.696116 10955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:12.703604 10955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45369
I20260928 23:48:12.703737 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45369 every 8 connection(s)
I20260928 23:48:12.704646 10955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:12.705952 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 10955
I20260928 23:48:12.706108 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:12.706446 21953 external_mini_cluster.cc:1428] Setting key 6e6a586e04b23925c3949d623f57d14c
I20260928 23:48:12.707966 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:12.712368 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:12.707944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:56125 (local address 127.0.0.1:45369)
0928 23:48:12.708403 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:12.708422 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:12.708464 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:48:12.709302 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:12.709323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:12.709755 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:12.710142 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:12.710916 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:12.710922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:12.710938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:12.710949 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:12.711021 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:12.711048 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:12.711192 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:12.711262 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 23:48:12.711434 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":157,"threads_started":1}
I20260928 23:48:12.715068 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4: Bootstrap starting.
I20260928 23:48:12.715473 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:12.717417 11023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:12.718235 11023 log.cc:824] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:12.719424 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:12.717658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:49537 (local address 127.0.0.1:45369)
0928 23:48:12.717798 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:12.717804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:12.717982 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:48:12.718137 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:12.718141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:12.718182 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:12.718270 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:12.719040 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:12.719044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:12.719046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:12.719058 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:12.719091 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:12.719102 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:12.719182 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:12.719290 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:48:12.719330 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 23:48:12.720530 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4: No bootstrap required, opened a new log
I20260928 23:48:12.724074 11023 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260928 23:48:12.724349 11023 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:12.724442 11023 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd8bd3ae8804d8fb4de7897552c3ec4, State: Initialized, Role: FOLLOWER
I20260928 23:48:12.724898 11023 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260928 23:48:12.725057 11023 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:12.725145 11023 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:12.725268 11023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:12.726399 11023 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260928 23:48:12.726778 11023 leader_election.cc:302] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd8bd3ae8804d8fb4de7897552c3ec4; no voters:
I20260928 23:48:12.727126 11023 leader_election.cc:288] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:12.727321 11033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:12.727640 11033 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [term 1 LEADER]: Becoming Leader. State: Replica: ccd8bd3ae8804d8fb4de7897552c3ec4, State: Running, Role: LEADER
I20260928 23:48:12.728004 11033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260928 23:48:12.728495 11023 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:12.730150 11034 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260928 23:48:12.730280 11034 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:12.730224 11035 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd8bd3ae8804d8fb4de7897552c3ec4. Latest consensus state: current_term: 1 leader_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd8bd3ae8804d8fb4de7897552c3ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260928 23:48:12.730336 11035 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:12.730688 11042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:12.734063 11042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:12.740249 11042 catalog_manager.cc:1347] Generated new cluster ID: 8d8816743d0f43c890df3cc4fa6bef25
I20260928 23:48:12.740341 11042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:12.750466 11042 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:12.750805 11042 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:12.767153 11042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd8bd3ae8804d8fb4de7897552c3ec4: Generated new TSK 0
I20260928 23:48:12.768074 11042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:12.778038 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 10955
2026-09-28T23:48:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:12 dist-test-slave-c3s8 krb5kdc[11054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:12 dist-test-slave-c3s8 krb5kdc[11054](info): set up 2 sockets
Sep 28 23:48:12 dist-test-slave-c3s8 krb5kdc[11054](info): commencing operation
krb5kdc: starting...
W20260928 23:48:13.846482 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:13 dist-test-slave-c3s8 krb5kdc[11054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:13.984894 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37757
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:37757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:14.104432 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:14.104761 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:14.104843 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:14.108958 11067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:14.109061 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:14.109100 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:14.109135 11067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:14.109174 11067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:14.114712 11067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:37757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:37757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:14.115863 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:14.116962 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:14.126310 11073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:14.126310 11075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:14.126500 11067 server_base.cc:1034] running on GCE node
W20260928 23:48:14.126315 11072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:14.127252 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:14.128419 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:14.129642 11067 hybrid_clock.cc:648] HybridClock initialized: now 1790639294129622 us; error 69 us; skew 500 ppm
Sep 28 23:48:14 dist-test-slave-c3s8 krb5kdc[11054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639294, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:14.132661 11067 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:14.133679 11067 webserver.cc:460] Webserver started at http://127.21.112.126:34869/ using document root <none> and password file <none>
I20260928 23:48:14.134291 11067 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:14.134397 11067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:14.134635 11067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:14.136746 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "678e30ffa0e9405598f52a6c8e5b32c0"
format_stamp: "Formatted at 2026-09-28 23:48:14 on dist-test-slave-c3s8"
server_key: "5601c7c0c9da9bbaab7ac97528784544"
server_key_iv: "c2a4a7983c39e6c7b263b97a37371a43"
server_key_version: "encryptionkey@0"
I20260928 23:48:14.137321 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "678e30ffa0e9405598f52a6c8e5b32c0"
format_stamp: "Formatted at 2026-09-28 23:48:14 on dist-test-slave-c3s8"
server_key: "5601c7c0c9da9bbaab7ac97528784544"
server_key_iv: "c2a4a7983c39e6c7b263b97a37371a43"
server_key_version: "encryptionkey@0"
I20260928 23:48:14.141254 11067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 23:48:14.143924 11082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:14.145295 11067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:48:14.145496 11067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "678e30ffa0e9405598f52a6c8e5b32c0"
format_stamp: "Formatted at 2026-09-28 23:48:14 on dist-test-slave-c3s8"
server_key: "5601c7c0c9da9bbaab7ac97528784544"
server_key_iv: "c2a4a7983c39e6c7b263b97a37371a43"
server_key_version: "encryptionkey@0"
I20260928 23:48:14.145699 11067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:14.158191 11067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:14.161958 11067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:14.162186 11067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:14.170061 11067 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:37757
I20260928 23:48:14.170162 11134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:37757 every 8 connection(s)
I20260928 23:48:14.171212 11067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:14.172299 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11067
I20260928 23:48:14.172504 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:14.172818 21953 external_mini_cluster.cc:1428] Setting key 7c2bedeae3f0b1908150e35f02526f6e
I20260928 23:48:14.175025 11135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:14.182513 11135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0: Bootstrap starting.
Sep 28 23:48:14 dist-test-slave-c3s8 krb5kdc[11054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:14.185613 11135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:14.186574 11135 log.cc:824] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:14.189288 11135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0: No bootstrap required, opened a new log
I20260928 23:48:14.190074 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:14.174345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45783 (local address 127.21.112.126:37757)
0928 23:48:14.174781 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:14.174800 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:14.174845 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:48:14.175731 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:14.175756 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:14.176206 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:14.176627 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:14.177380 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:14.178978 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:14.179912 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:14.180661 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:14.180864 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:14.184078 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:14.184084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:14.184101 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:14.184140 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:14.186283 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:14.187059 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:14.187065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:14.187071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:14.187195 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:14.187708 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:14.187714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:14.187718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:14.188261 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:14.188534 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:14.188814 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 23:48:14.189071 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260928 23:48:14.193372 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:14.193841 11135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37757 } }
I20260928 23:48:14.194072 11135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:14.194144 11135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678e30ffa0e9405598f52a6c8e5b32c0, State: Initialized, Role: FOLLOWER
I20260928 23:48:14.194674 11135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37757 } }
I20260928 23:48:14.194844 11135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:14.194916 11135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:14.195015 11135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:14.196327 11135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37757 } }
I20260928 23:48:14.196754 11135 leader_election.cc:302] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 678e30ffa0e9405598f52a6c8e5b32c0; no voters:
I20260928 23:48:14.197189 11135 leader_election.cc:288] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:14.197257 11145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:14.197443 11145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [term 1 LEADER]: Becoming Leader. State: Replica: 678e30ffa0e9405598f52a6c8e5b32c0, State: Running, Role: LEADER
I20260928 23:48:14.197849 11145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37757 } }
I20260928 23:48:14.198565 11135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:14.199580 11147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37757 } } }
I20260928 23:48:14.199723 11147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:14.200141 11154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:14.200042 11146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 678e30ffa0e9405598f52a6c8e5b32c0. Latest consensus state: current_term: 1 leader_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "678e30ffa0e9405598f52a6c8e5b32c0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 37757 } } }
I20260928 23:48:14.200505 11146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:14.203540 11154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:14.205380 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:14.195659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38491 (local address 127.21.112.126:37757)
0928 23:48:14.195783 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:14.195787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:14.195936 (+ 149us) server_negotiation.cc:407] Connection header received
0928 23:48:14.196070 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:14.196072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:14.196113 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:14.196181 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:14.197040 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:14.197970 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:14.199042 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:14.199444 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:14.199555 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:14.201582 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:14.201585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:14.201588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:14.201615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:14.203398 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:14.203969 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:14.203972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:14.203974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:14.204062 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:14.204491 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:14.204495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:14.204497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:14.204684 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:14.204855 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:14.205130 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 23:48:14.205226 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:48:14.213881 11154 catalog_manager.cc:1347] Generated new cluster ID: 1210443f85a04dc3810715a7f62f4f5e
I20260928 23:48:14.214027 11154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:14.222023 11154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:14.223174 11154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:14.238538 11154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 678e30ffa0e9405598f52a6c8e5b32c0: Generated new TSK 0
I20260928 23:48:14.239392 11154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:14.245680 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11067
W20260928 23:48:14.250525 11052 connection.cc:535] client connection to 127.21.112.126:37757 recv error: Network error: failed to read from TLS socket (remote: 127.21.112.126:37757): Connection reset by peer (error 104)
2026-09-28T23:48:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:14 dist-test-slave-c3s8 krb5kdc[11166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:14 dist-test-slave-c3s8 krb5kdc[11166](info): set up 2 sockets
Sep 28 23:48:14 dist-test-slave-c3s8 krb5kdc[11166](info): commencing operation
krb5kdc: starting...
W20260928 23:48:15.302306 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:15 dist-test-slave-c3s8 krb5kdc[11166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:15.433807 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34019
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:15.549224 11179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:15.549504 11179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:15.549615 11179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:15.553973 11179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:15.554045 11179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:15.554065 11179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:15.554083 11179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:15.554098 11179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:15.559784 11179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:34019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:15.560951 11179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:15.562125 11179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:15.571235 11184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:15.571363 11179 server_base.cc:1034] running on GCE node
W20260928 23:48:15.571235 11187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:15.571264 11185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:15.572146 11179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:15.573325 11179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:15.574542 11179 hybrid_clock.cc:648] HybridClock initialized: now 1790639295574529 us; error 30 us; skew 500 ppm
Sep 28 23:48:15 dist-test-slave-c3s8 krb5kdc[11166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639295, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:15.577879 11179 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:15.578858 11179 webserver.cc:460] Webserver started at http://127.21.112.126:36073/ using document root <none> and password file <none>
I20260928 23:48:15.579550 11179 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:15.579643 11179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:15.579892 11179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:15.582227 11179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "2edcd1b1d15c4885859007b2c42e3d21"
format_stamp: "Formatted at 2026-09-28 23:48:15 on dist-test-slave-c3s8"
server_key: "74f81e262b6be5c64b5f28f4aa1a63e3"
server_key_iv: "c3b5ecb34040ba80229fea28dc7b909a"
server_key_version: "encryptionkey@0"
I20260928 23:48:15.582800 11179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "2edcd1b1d15c4885859007b2c42e3d21"
format_stamp: "Formatted at 2026-09-28 23:48:15 on dist-test-slave-c3s8"
server_key: "74f81e262b6be5c64b5f28f4aa1a63e3"
server_key_iv: "c3b5ecb34040ba80229fea28dc7b909a"
server_key_version: "encryptionkey@0"
I20260928 23:48:15.586732 11179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 23:48:15.589511 11194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:15.590817 11179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 23:48:15.590991 11179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "2edcd1b1d15c4885859007b2c42e3d21"
format_stamp: "Formatted at 2026-09-28 23:48:15 on dist-test-slave-c3s8"
server_key: "74f81e262b6be5c64b5f28f4aa1a63e3"
server_key_iv: "c3b5ecb34040ba80229fea28dc7b909a"
server_key_version: "encryptionkey@0"
I20260928 23:48:15.591166 11179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:15.603988 11179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:15.607357 11179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:15.607613 11179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:15.614698 11179 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:34019
I20260928 23:48:15.614713 11246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:34019 every 8 connection(s)
I20260928 23:48:15.615716 11179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:15.618660 11247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:15.620332 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11179
I20260928 23:48:15.620554 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:15.620843 21953 external_mini_cluster.cc:1428] Setting key 5ed2340c0141cfec617502de803049c9
I20260928 23:48:15.625022 11247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21: Bootstrap starting.
I20260928 23:48:15.627720 11247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:15.628440 11247 log.cc:824] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:15.630388 11247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21: No bootstrap required, opened a new log
I20260928 23:48:15.630450 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:15.622228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54845 (local address 127.21.112.126:34019)
0928 23:48:15.622637 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:15.622653 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:15.622690 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:48:15.623328 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:15.623345 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:15.623719 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:15.624013 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:15.624354 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:15.625366 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:15.626616 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:15.627113 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:15.627241 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:15.628323 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:15.628331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:15.628350 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:15.628847 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:15.629070 (+ 223us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:15.629108 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:15.629280 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:15.629519 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:48:15.629721 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":142,"threads_started":1}
I20260928 23:48:15.633410 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:15.634491 11247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcd1b1d15c4885859007b2c42e3d21" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34019 } }
I20260928 23:48:15.634809 11247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:15.634869 11247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edcd1b1d15c4885859007b2c42e3d21, State: Initialized, Role: FOLLOWER
I20260928 23:48:15.635357 11247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcd1b1d15c4885859007b2c42e3d21" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34019 } }
I20260928 23:48:15.635524 11247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:15.635602 11247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:15.635694 11247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:15.636929 11247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcd1b1d15c4885859007b2c42e3d21" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34019 } }
I20260928 23:48:15.637312 11247 leader_election.cc:302] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edcd1b1d15c4885859007b2c42e3d21; no voters:
I20260928 23:48:15.637656 11247 leader_election.cc:288] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:15.637764 11257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:15.637970 11257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [term 1 LEADER]: Becoming Leader. State: Replica: 2edcd1b1d15c4885859007b2c42e3d21, State: Running, Role: LEADER
I20260928 23:48:15.638293 11257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcd1b1d15c4885859007b2c42e3d21" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34019 } }
I20260928 23:48:15.639050 11247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:15.640179 11258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2edcd1b1d15c4885859007b2c42e3d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcd1b1d15c4885859007b2c42e3d21" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34019 } } }
I20260928 23:48:15.640241 11259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edcd1b1d15c4885859007b2c42e3d21. Latest consensus state: current_term: 1 leader_uuid: "2edcd1b1d15c4885859007b2c42e3d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edcd1b1d15c4885859007b2c42e3d21" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 34019 } } }
I20260928 23:48:15.640360 11259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:15.640306 11258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:15.640684 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:15.635572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35859 (local address 127.21.112.126:34019)
0928 23:48:15.635698 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:15.635703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:15.635945 (+ 242us) server_negotiation.cc:407] Connection header received
0928 23:48:15.636027 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:15.636033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:15.636082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:15.636163 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:15.636387 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:15.637155 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:15.638389 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:15.638738 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:15.638844 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:15.639755 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:15.639759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:15.639763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:15.639781 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:15.639814 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:15.639828 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:15.639937 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:15.640492 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 23:48:15.640538 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:48:15.640874 11266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:15.644622 11266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:15.650841 11266 catalog_manager.cc:1347] Generated new cluster ID: c60493b4b67b467fa80b5d688c41b6ce
I20260928 23:48:15.650964 11266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:15.662068 11266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:15.663468 11266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:15.673812 11266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2edcd1b1d15c4885859007b2c42e3d21: Generated new TSK 0
I20260928 23:48:15.674685 11266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:15.683609 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11179
W20260928 23:48:15.691082 11164 connection.cc:535] client connection to 127.21.112.126:34019 recv error: Network error: failed to read from TLS socket (remote: 127.21.112.126:34019): Connection reset by peer (error 104)
2026-09-28T23:48:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:15 dist-test-slave-c3s8 krb5kdc[11278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:15 dist-test-slave-c3s8 krb5kdc[11278](info): set up 2 sockets
Sep 28 23:48:15 dist-test-slave-c3s8 krb5kdc[11278](info): commencing operation
krb5kdc: starting...
W20260928 23:48:16.750296 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:16 dist-test-slave-c3s8 krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:16.880689 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36287
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:16.993487 11291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:16.993893 11291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:16.994141 11291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:16.998404 11291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:16.998508 11291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:16.998558 11291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:16.998596 11291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:16.998626 11291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:17.004129 11291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:36287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:17.005271 11291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:17.006349 11291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:17.015470 11299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:17.015609 11291 server_base.cc:1034] running on GCE node
W20260928 23:48:17.015515 11297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:17.015793 11296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:17.016362 11291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:17.017655 11291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:17.018850 11291 hybrid_clock.cc:648] HybridClock initialized: now 1790639297018836 us; error 33 us; skew 500 ppm
Sep 28 23:48:17 dist-test-slave-c3s8 krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639297, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:17.031459 11291 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:17.032550 11291 webserver.cc:460] Webserver started at http://127.21.112.126:41837/ using document root <none> and password file <none>
I20260928 23:48:17.033162 11291 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:17.033260 11291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:17.033511 11291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:17.035704 11291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "9da71381e0164eefb786710101330b87"
format_stamp: "Formatted at 2026-09-28 23:48:17 on dist-test-slave-c3s8"
server_key: "863fb3af621bd466ba192d255566744f"
server_key_iv: "eac5a92a212c9c45cc43f2154b1149fb"
server_key_version: "encryptionkey@0"
I20260928 23:48:17.036262 11291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "9da71381e0164eefb786710101330b87"
format_stamp: "Formatted at 2026-09-28 23:48:17 on dist-test-slave-c3s8"
server_key: "863fb3af621bd466ba192d255566744f"
server_key_iv: "eac5a92a212c9c45cc43f2154b1149fb"
server_key_version: "encryptionkey@0"
I20260928 23:48:17.040446 11291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 23:48:17.043378 11306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:17.044775 11291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:48:17.044965 11291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "9da71381e0164eefb786710101330b87"
format_stamp: "Formatted at 2026-09-28 23:48:17 on dist-test-slave-c3s8"
server_key: "863fb3af621bd466ba192d255566744f"
server_key_iv: "eac5a92a212c9c45cc43f2154b1149fb"
server_key_version: "encryptionkey@0"
I20260928 23:48:17.045099 11291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:17.050137 11291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:17.053404 11291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:17.053742 11291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:17.061321 11291 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:36287
I20260928 23:48:17.061375 11358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:36287 every 8 connection(s)
I20260928 23:48:17.062340 11291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:17.065202 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:17.067885 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11291
I20260928 23:48:17.068023 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:17.068338 21953 external_mini_cluster.cc:1428] Setting key ac1599854831fe4c9033070f7f4c5e65
I20260928 23:48:17.071528 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87: Bootstrap starting.
I20260928 23:48:17.073526 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:17.069781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53545 (local address 127.21.112.126:36287)
0928 23:48:17.070246 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:17.070260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:17.070297 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:48:17.070927 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:17.070945 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:17.071230 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:17.071571 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:17.072350 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:17.072355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:17.072367 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:17.072377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:17.072430 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:17.072449 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:17.072595 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:17.072667 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:48:17.072834 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260928 23:48:17.074120 11359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:17.074860 11359 log.cc:824] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:17.076486 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:17.076928 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87: No bootstrap required, opened a new log
I20260928 23:48:17.080466 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:17.078592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47235 (local address 127.21.112.126:36287)
0928 23:48:17.078727 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:17.078731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:17.078931 (+ 200us) server_negotiation.cc:407] Connection header received
0928 23:48:17.079055 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:17.079057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:17.079084 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:17.079149 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:17.080047 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:17.080050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:17.080051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:17.080061 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:17.080085 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:17.080095 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:17.080170 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:17.080295 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:48:17.080326 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:48:17.081409 11359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da71381e0164eefb786710101330b87" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36287 } }
I20260928 23:48:17.081720 11359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:17.081781 11359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da71381e0164eefb786710101330b87, State: Initialized, Role: FOLLOWER
I20260928 23:48:17.082180 11359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da71381e0164eefb786710101330b87" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36287 } }
I20260928 23:48:17.082333 11359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:17.082428 11359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:17.082531 11359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:17.083454 11359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da71381e0164eefb786710101330b87" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36287 } }
I20260928 23:48:17.083794 11359 leader_election.cc:302] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9da71381e0164eefb786710101330b87; no voters:
I20260928 23:48:17.084110 11359 leader_election.cc:288] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:17.084209 11369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:17.084414 11369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [term 1 LEADER]: Becoming Leader. State: Replica: 9da71381e0164eefb786710101330b87, State: Running, Role: LEADER
I20260928 23:48:17.084851 11369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da71381e0164eefb786710101330b87" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36287 } }
I20260928 23:48:17.085318 11359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:17.086521 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9da71381e0164eefb786710101330b87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da71381e0164eefb786710101330b87" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36287 } } }
I20260928 23:48:17.086647 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:17.086985 11371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9da71381e0164eefb786710101330b87. Latest consensus state: current_term: 1 leader_uuid: "9da71381e0164eefb786710101330b87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9da71381e0164eefb786710101330b87" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 36287 } } }
I20260928 23:48:17.087067 11378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:17.087139 11371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:17.090065 11378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:17.095539 11378 catalog_manager.cc:1347] Generated new cluster ID: db4dbe1406d64a518245e43e90373faa
I20260928 23:48:17.095616 11378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:17.112815 11378 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:17.113201 11378 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:17.125945 11378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9da71381e0164eefb786710101330b87: Generated new TSK 0
I20260928 23:48:17.126825 11378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:17.137691 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11291
2026-09-28T23:48:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:17 dist-test-slave-c3s8 krb5kdc[11390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:17 dist-test-slave-c3s8 krb5kdc[11390](info): set up 2 sockets
Sep 28 23:48:17 dist-test-slave-c3s8 krb5kdc[11390](info): commencing operation
krb5kdc: starting...
W20260928 23:48:18.226404 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:18 dist-test-slave-c3s8 krb5kdc[11390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:18.356304 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:42429
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:42429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:18.472971 11403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:18.473272 11403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:18.473317 11403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:18.477368 11403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:18.477432 11403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:18.477452 11403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:18.477469 11403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:18.477485 11403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:18.482918 11403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:42429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:42429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:18.483924 11403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:18.484891 11403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:18.493440 11408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:18.493450 11411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:18.493618 11409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:18.493917 11403 server_base.cc:1034] running on GCE node
I20260928 23:48:18.494540 11403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:18.495640 11403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:18.496843 11403 hybrid_clock.cc:648] HybridClock initialized: now 1790639298496823 us; error 33 us; skew 500 ppm
Sep 28 23:48:18 dist-test-slave-c3s8 krb5kdc[11390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:18.499758 11403 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:18.500712 11403 webserver.cc:460] Webserver started at http://127.21.112.126:44623/ using document root <none> and password file <none>
I20260928 23:48:18.501293 11403 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:18.501355 11403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:18.501679 11403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:18.503805 11403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "a9034eb01f724bafb6b7dcb8358a5fb0"
format_stamp: "Formatted at 2026-09-28 23:48:18 on dist-test-slave-c3s8"
server_key: "d60c272b35bd5a58d55c27caf457856b"
server_key_iv: "ad045d96fb06d20e20ad192c77c51dd7"
server_key_version: "encryptionkey@0"
I20260928 23:48:18.504398 11403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "a9034eb01f724bafb6b7dcb8358a5fb0"
format_stamp: "Formatted at 2026-09-28 23:48:18 on dist-test-slave-c3s8"
server_key: "d60c272b35bd5a58d55c27caf457856b"
server_key_iv: "ad045d96fb06d20e20ad192c77c51dd7"
server_key_version: "encryptionkey@0"
I20260928 23:48:18.508359 11403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 23:48:18.510929 11418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:18.512316 11403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:48:18.512458 11403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "a9034eb01f724bafb6b7dcb8358a5fb0"
format_stamp: "Formatted at 2026-09-28 23:48:18 on dist-test-slave-c3s8"
server_key: "d60c272b35bd5a58d55c27caf457856b"
server_key_iv: "ad045d96fb06d20e20ad192c77c51dd7"
server_key_version: "encryptionkey@0"
I20260928 23:48:18.512575 11403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:18.536923 11403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:18.540609 11403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:18.540830 11403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:18.548471 11403 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:42429
I20260928 23:48:18.548489 11470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:42429 every 8 connection(s)
I20260928 23:48:18.549458 11403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:18.552268 11471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:18.553654 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11403
I20260928 23:48:18.553786 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:18.554118 21953 external_mini_cluster.cc:1428] Setting key fc260d011f977072ff760de0de7daf41
I20260928 23:48:18.558609 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0: Bootstrap starting.
I20260928 23:48:18.560781 11471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:18.561642 11471 log.cc:824] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0: Log is configured to *not* fsync() on all Append() calls
Sep 28 23:48:18 dist-test-slave-c3s8 krb5kdc[11390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:18.563674 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0: No bootstrap required, opened a new log
I20260928 23:48:18.567153 11471 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42429 } }
I20260928 23:48:18.567408 11471 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:18.567497 11471 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9034eb01f724bafb6b7dcb8358a5fb0, State: Initialized, Role: FOLLOWER
I20260928 23:48:18.567988 11471 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42429 } }
I20260928 23:48:18.568138 11471 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:18.568226 11471 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:18.568332 11471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:18.568581 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:18.555568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47713 (local address 127.21.112.126:42429)
0928 23:48:18.556005 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:18.556020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:18.556056 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:48:18.556749 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:18.556767 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:18.557111 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:18.557471 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:18.558316 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:18.559352 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:18.560225 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:18.560702 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:18.560889 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:18.563721 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:18.563726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:18.563738 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:18.563788 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:18.565530 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:18.566024 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:18.566037 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:18.566042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:18.566153 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:18.566442 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:18.566446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:18.566447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:18.566837 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:18.567064 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:18.567311 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 23:48:18.567654 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260928 23:48:18.569507 11471 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42429 } }
I20260928 23:48:18.569911 11471 leader_election.cc:302] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9034eb01f724bafb6b7dcb8358a5fb0; no voters:
I20260928 23:48:18.570233 11471 leader_election.cc:288] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:18.570340 11476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:18.570509 11476 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [term 1 LEADER]: Becoming Leader. State: Replica: a9034eb01f724bafb6b7dcb8358a5fb0, State: Running, Role: LEADER
I20260928 23:48:18.571099 11476 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42429 } }
I20260928 23:48:18.571504 11471 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:18.571725 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:18.572924 11477 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42429 } } }
I20260928 23:48:18.573056 11477 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:18.573349 11476 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9034eb01f724bafb6b7dcb8358a5fb0. Latest consensus state: current_term: 1 leader_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9034eb01f724bafb6b7dcb8358a5fb0" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 42429 } } }
I20260928 23:48:18.573451 11476 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0 [sys.catalog]: This master's current role is: LEADER
W20260928 23:48:18.576349 11495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:18.576427 11495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:18.576534 11493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:18.577219 11493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:18.583196 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:18.574268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53071 (local address 127.21.112.126:42429)
0928 23:48:18.574394 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:18.574399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:18.574410 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:48:18.574450 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:18.574452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:18.574484 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:18.574567 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:18.575882 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:18.576484 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:18.577618 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:18.577977 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:18.578081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:18.579735 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:18.579739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:18.579742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:18.579767 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:18.581480 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:18.581950 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:18.581954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:18.581956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:18.582048 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:18.582340 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:18.582345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:18.582347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:18.582528 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:18.582668 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:18.582912 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 23:48:18.583025 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 23:48:18.583722 11493 catalog_manager.cc:1347] Generated new cluster ID: f5340fe1b84b47a5b4eda60cd1252fb9
I20260928 23:48:18.583777 11493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:18.600876 11493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:18.602089 11493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:18.611114 11493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9034eb01f724bafb6b7dcb8358a5fb0: Generated new TSK 0
I20260928 23:48:18.612102 11493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:18.624728 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11403
2026-09-28T23:48:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:18 dist-test-slave-c3s8 krb5kdc[11502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:18 dist-test-slave-c3s8 krb5kdc[11502](info): set up 2 sockets
Sep 28 23:48:18 dist-test-slave-c3s8 krb5kdc[11502](info): commencing operation
krb5kdc: starting...
W20260928 23:48:19.682335 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:19 dist-test-slave-c3s8 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:19.812476 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41783
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:19.924641 11515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:19.924989 11515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:19.925068 11515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:19.930796 11515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:19.930946 11515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:19.931044 11515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:19.931095 11515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:19.931144 11515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:19.938148 11515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:19.939226 11515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:19.940300 11515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:19.950133 11523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:19.950099 11521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:19.950129 11520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:19.950186 11515 server_base.cc:1034] running on GCE node
I20260928 23:48:19.951144 11515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:19.952450 11515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:19.953636 11515 hybrid_clock.cc:648] HybridClock initialized: now 1790639299953620 us; error 32 us; skew 500 ppm
Sep 28 23:48:19 dist-test-slave-c3s8 krb5kdc[11502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:19.956660 11515 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:19.957753 11515 webserver.cc:460] Webserver started at http://127.21.112.126:36933/ using document root <none> and password file <none>
I20260928 23:48:19.958370 11515 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:19.958465 11515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:19.958706 11515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:19.960834 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "49c12607cd20468386af515f1f525df8"
format_stamp: "Formatted at 2026-09-28 23:48:19 on dist-test-slave-c3s8"
server_key: "cbea452dd20fe5cf5aad894a08f0edfb"
server_key_iv: "d1353ea9603b735db797a18d6fc47e69"
server_key_version: "encryptionkey@0"
I20260928 23:48:19.961424 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "49c12607cd20468386af515f1f525df8"
format_stamp: "Formatted at 2026-09-28 23:48:19 on dist-test-slave-c3s8"
server_key: "cbea452dd20fe5cf5aad894a08f0edfb"
server_key_iv: "d1353ea9603b735db797a18d6fc47e69"
server_key_version: "encryptionkey@0"
I20260928 23:48:19.965340 11515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 23:48:19.967985 11530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:19.969419 11515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 23:48:19.969612 11515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "49c12607cd20468386af515f1f525df8"
format_stamp: "Formatted at 2026-09-28 23:48:19 on dist-test-slave-c3s8"
server_key: "cbea452dd20fe5cf5aad894a08f0edfb"
server_key_iv: "d1353ea9603b735db797a18d6fc47e69"
server_key_version: "encryptionkey@0"
I20260928 23:48:19.969748 11515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:19.991182 11515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:19.995122 11515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:19.995342 11515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:20.002790 11515 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:41783
I20260928 23:48:20.002810 11582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:41783 every 8 connection(s)
I20260928 23:48:20.003779 11515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:20.006639 11583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:20.009502 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11515
I20260928 23:48:20.009673 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:20.010000 21953 external_mini_cluster.cc:1428] Setting key e1c06f07f825cfe57087a36022dac7d1
I20260928 23:48:20.013131 11583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8: Bootstrap starting.
I20260928 23:48:20.016073 11583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:20.017052 11583 log.cc:824] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:20.019568 11583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8: No bootstrap required, opened a new log
I20260928 23:48:20.022539 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:20.011546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46175 (local address 127.21.112.126:41783)
0928 23:48:20.013489 (+ 1943us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:20.013509 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:20.013578 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:48:20.014475 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:20.014503 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:20.014965 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:20.015353 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:20.015934 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:20.017520 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:20.018655 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:20.019166 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:20.019353 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:20.020395 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:20.020402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:20.020430 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:20.020951 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:20.021031 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:20.021070 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:20.021173 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:20.021406 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 23:48:20.021697 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1764,"thread_start_us":143,"threads_started":1}
I20260928 23:48:20.023818 11583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c12607cd20468386af515f1f525df8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41783 } }
I20260928 23:48:20.024096 11583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:20.024144 11583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c12607cd20468386af515f1f525df8, State: Initialized, Role: FOLLOWER
I20260928 23:48:20.024565 11583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c12607cd20468386af515f1f525df8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41783 } }
I20260928 23:48:20.024747 11583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:20.024816 11583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:20.024926 11583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:20.025650 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:20.026158 11583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c12607cd20468386af515f1f525df8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41783 } }
I20260928 23:48:20.026561 11583 leader_election.cc:302] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c12607cd20468386af515f1f525df8; no voters:
I20260928 23:48:20.026929 11583 leader_election.cc:288] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:20.027050 11588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:20.027349 11588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [term 1 LEADER]: Becoming Leader. State: Replica: 49c12607cd20468386af515f1f525df8, State: Running, Role: LEADER
I20260928 23:48:20.027726 11588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c12607cd20468386af515f1f525df8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41783 } }
I20260928 23:48:20.028352 11583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:20.028987 11588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c12607cd20468386af515f1f525df8. Latest consensus state: current_term: 1 leader_uuid: "49c12607cd20468386af515f1f525df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c12607cd20468386af515f1f525df8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41783 } } }
I20260928 23:48:20.029104 11588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:20.029475 11600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:20.029497 11593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c12607cd20468386af515f1f525df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c12607cd20468386af515f1f525df8" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41783 } } }
I20260928 23:48:20.029645 11593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:20.032678 11600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:20.035156 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:20.027927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50679 (local address 127.21.112.126:41783)
0928 23:48:20.028033 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:20.028038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:20.030814 (+ 2776us) server_negotiation.cc:407] Connection header received
0928 23:48:20.031043 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:20.031046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:20.031113 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:20.031229 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:20.031509 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:20.032028 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:20.033320 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:20.033636 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:20.033709 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:20.034690 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:20.034694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:20.034697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:20.034710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:20.034746 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:20.034757 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:20.034838 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:20.034931 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:48:20.034986 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:48:20.039008 11600 catalog_manager.cc:1347] Generated new cluster ID: 0a21b8d7c1fc4441ab9a367ea0e9efbc
I20260928 23:48:20.039083 11600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:20.052964 11600 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:20.054318 11600 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:20.060578 11600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49c12607cd20468386af515f1f525df8: Generated new TSK 0
I20260928 23:48:20.061414 11600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:20.070138 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11515
2026-09-28T23:48:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:20 dist-test-slave-c3s8 krb5kdc[11614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:20 dist-test-slave-c3s8 krb5kdc[11614](info): set up 2 sockets
Sep 28 23:48:20 dist-test-slave-c3s8 krb5kdc[11614](info): commencing operation
krb5kdc: starting...
W20260928 23:48:21.142280 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:21 dist-test-slave-c3s8 krb5kdc[11614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:21.270745 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41893
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:41893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:21.384346 11627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:21.384680 11627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:21.384755 11627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:21.388923 11627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:21.389024 11627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:21.389061 11627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:21.389123 11627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:21.389163 11627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:21.394605 11627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:41893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:41893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:21.395768 11627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:21.396849 11627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:21.405761 11635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:21.405762 11633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:21.405766 11632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:21.406065 11627 server_base.cc:1034] running on GCE node
I20260928 23:48:21.406756 11627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:21.407894 11627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:21.409106 11627 hybrid_clock.cc:648] HybridClock initialized: now 1790639301409066 us; error 46 us; skew 500 ppm
Sep 28 23:48:21 dist-test-slave-c3s8 krb5kdc[11614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639301, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:21.412209 11627 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:21.413192 11627 webserver.cc:460] Webserver started at http://127.21.112.126:34827/ using document root <none> and password file <none>
I20260928 23:48:21.413889 11627 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:21.413985 11627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:21.414249 11627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:21.416420 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "7efd731143694cd48603bfc58466f20a"
format_stamp: "Formatted at 2026-09-28 23:48:21 on dist-test-slave-c3s8"
server_key: "fe892cdae50e06f1fb4a94672ad40592"
server_key_iv: "1a4f3bfde82cbb0cca0c3e914af05651"
server_key_version: "encryptionkey@0"
I20260928 23:48:21.417076 11627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "7efd731143694cd48603bfc58466f20a"
format_stamp: "Formatted at 2026-09-28 23:48:21 on dist-test-slave-c3s8"
server_key: "fe892cdae50e06f1fb4a94672ad40592"
server_key_iv: "1a4f3bfde82cbb0cca0c3e914af05651"
server_key_version: "encryptionkey@0"
I20260928 23:48:21.421118 11627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 23:48:21.423728 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:21.425132 11627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 23:48:21.425336 11627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "7efd731143694cd48603bfc58466f20a"
format_stamp: "Formatted at 2026-09-28 23:48:21 on dist-test-slave-c3s8"
server_key: "fe892cdae50e06f1fb4a94672ad40592"
server_key_iv: "1a4f3bfde82cbb0cca0c3e914af05651"
server_key_version: "encryptionkey@0"
I20260928 23:48:21.425468 11627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:21.430408 11627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:21.433836 11627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:21.434039 11627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:21.441213 11627 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:41893
I20260928 23:48:21.441269 11694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:41893 every 8 connection(s)
I20260928 23:48:21.442337 11627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:21.445178 11695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:21.447695 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11627
I20260928 23:48:21.447862 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:21.448144 21953 external_mini_cluster.cc:1428] Setting key d4a306f0cf242cdbd160be4d00fe2fb8
I20260928 23:48:21.451865 11695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a: Bootstrap starting.
I20260928 23:48:21.454254 11698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:21.449752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45607 (local address 127.21.112.126:41893)
0928 23:48:21.450224 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:21.450243 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:21.450289 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:48:21.451123 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:21.451147 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:21.451517 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:21.451903 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:21.452796 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:21.452802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:21.452817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:21.452830 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:21.452896 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:21.452924 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:21.453064 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:21.453133 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:48:21.453284 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260928 23:48:21.454972 11695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:21.455868 11695 log.cc:824] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:21.457644 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:21.458220 11695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a: No bootstrap required, opened a new log
I20260928 23:48:21.461601 11698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:21.459800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52949 (local address 127.21.112.126:41893)
0928 23:48:21.459932 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:21.459937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:21.460120 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:48:21.460268 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:21.460271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:21.460304 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:21.460399 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:21.461185 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:21.461189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:21.461191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:48:21.461203 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:21.461231 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:48:21.461250 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:21.461342 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:21.461415 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 23:48:21.461455 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:48:21.462402 11695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd731143694cd48603bfc58466f20a" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41893 } }
I20260928 23:48:21.462707 11695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:48:21.462786 11695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efd731143694cd48603bfc58466f20a, State: Initialized, Role: FOLLOWER
I20260928 23:48:21.463230 11695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd731143694cd48603bfc58466f20a" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41893 } }
I20260928 23:48:21.463383 11695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:48:21.463469 11695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:48:21.463585 11695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:21.464763 11695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd731143694cd48603bfc58466f20a" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41893 } }
I20260928 23:48:21.465116 11695 leader_election.cc:302] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7efd731143694cd48603bfc58466f20a; no voters:
I20260928 23:48:21.465447 11695 leader_election.cc:288] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:48:21.465611 11705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:21.465885 11705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [term 1 LEADER]: Becoming Leader. State: Replica: 7efd731143694cd48603bfc58466f20a, State: Running, Role: LEADER
I20260928 23:48:21.466308 11705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd731143694cd48603bfc58466f20a" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41893 } }
I20260928 23:48:21.466732 11695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:21.468058 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7efd731143694cd48603bfc58466f20a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd731143694cd48603bfc58466f20a" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41893 } } }
I20260928 23:48:21.468194 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:21.468567 11707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7efd731143694cd48603bfc58466f20a. Latest consensus state: current_term: 1 leader_uuid: "7efd731143694cd48603bfc58466f20a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efd731143694cd48603bfc58466f20a" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 41893 } } }
I20260928 23:48:21.468667 11707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:21.468955 11717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:21.472409 11717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:21.478453 11717 catalog_manager.cc:1347] Generated new cluster ID: c84ab056d6a84a6cac1cd81433dc84aa
I20260928 23:48:21.478538 11717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:21.485301 11717 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:21.485714 11717 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:21.494709 11717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7efd731143694cd48603bfc58466f20a: Generated new TSK 0
I20260928 23:48:21.495637 11717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:48:21.508114 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11627
2026-09-28T23:48:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1437 ms)
[----------] 18 tests from AuthTokenIssuingTest (26156 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:21 dist-test-slave-c3s8 krb5kdc[11726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:21 dist-test-slave-c3s8 krb5kdc[11726](info): set up 2 sockets
Sep 28 23:48:21 dist-test-slave-c3s8 krb5kdc[11726](info): commencing operation
krb5kdc: starting...
W20260928 23:48:22.558321 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:22 dist-test-slave-c3s8 krb5kdc[11726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:22.693303 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43471
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43471,127.21.112.125:36699,127.21.112.124:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:22.810540 11739 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:48:22.810874 11739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:22.810954 11739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:22.811038 11739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:22.815214 11739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:22.815315 11739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:22.815353 11739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:22.815392 11739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:22.815428 11739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:22.820871 11739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43471,127.21.112.125:36699,127.21.112.124:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:22.822050 11739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:22.823082 11739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:22.831782 11747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:22.831805 11744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:22.831880 11739 server_base.cc:1034] running on GCE node
W20260928 23:48:22.831815 11745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:22.832793 11739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:22.833957 11739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:22.835165 11739 hybrid_clock.cc:648] HybridClock initialized: now 1790639302835150 us; error 26 us; skew 500 ppm
Sep 28 23:48:22 dist-test-slave-c3s8 krb5kdc[11726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:22.844971 11739 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:22.846177 11739 webserver.cc:460] Webserver started at http://127.21.112.126:39707/ using document root <none> and password file <none>
I20260928 23:48:22.846782 11739 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:22.846876 11739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:22.847128 11739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:22.849236 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "d03946032a9540ebada12617b3ac9e15"
format_stamp: "Formatted at 2026-09-28 23:48:22 on dist-test-slave-c3s8"
server_key: "08533eda2793409c5713aedff81b62a3"
server_key_iv: "ad74772ac93bf99cacc5eedcceb73e13"
server_key_version: "encryptionkey@0"
I20260928 23:48:22.849897 11739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "d03946032a9540ebada12617b3ac9e15"
format_stamp: "Formatted at 2026-09-28 23:48:22 on dist-test-slave-c3s8"
server_key: "08533eda2793409c5713aedff81b62a3"
server_key_iv: "ad74772ac93bf99cacc5eedcceb73e13"
server_key_version: "encryptionkey@0"
I20260928 23:48:22.853767 11739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:48:22.856331 11754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:22.857756 11739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:22.857940 11739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "d03946032a9540ebada12617b3ac9e15"
format_stamp: "Formatted at 2026-09-28 23:48:22 on dist-test-slave-c3s8"
server_key: "08533eda2793409c5713aedff81b62a3"
server_key_iv: "ad74772ac93bf99cacc5eedcceb73e13"
server_key_version: "encryptionkey@0"
I20260928 23:48:22.858059 11739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:22.875295 11739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:22.878944 11739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:22.879176 11739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:22.886622 11739 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:43471
I20260928 23:48:22.886947 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:43471 every 8 connection(s)
I20260928 23:48:22.887621 11739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:22.890403 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11739
I20260928 23:48:22.890439 11807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:22.890551 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:22.890825 21953 external_mini_cluster.cc:1428] Setting key 227914f00db96ab67d3984f5d2314889
I20260928 23:48:22.894407 11807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.112.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.125@KRBTEST.COM" created.
Sep 28 23:48:22 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:22.910909 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:22.896719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43471 (local address 127.0.0.1:42386)
0928 23:48:22.897218 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:22.897270 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:22.898207 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:22.899047 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:22.899052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:22.899370 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:22.899758 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:22.899776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:22.901132 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:22.901135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:22.902240 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:22.902250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:22.902968 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:22.902972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:22.903064 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:22.903802 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:22.903828 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:22.905716 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:22.907752 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:22.907762 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:22.907773 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:22.908010 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:22.908333 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:22.908338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:22.908340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:22.908463 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:22.909082 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:22.909089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:22.909237 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 23:48:22.909421 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 23:48:22.909849 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":107,"spinlock_wait_cycles":4864,"thread_start_us":147,"threads_started":1}
I20260928 23:48:22.910909 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:22.896719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.21.112.126:43471)
0928 23:48:22.897271 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:22.897289 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:22.897396 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:48:22.898551 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:22.898560 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:22.898836 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:22.899023 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:22.899939 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:22.900985 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:22.902390 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:22.902777 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:22.902973 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:22.905892 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:22.905898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:22.905910 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:22.905932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:22.907624 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:22.908134 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:22.908138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:22.908140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:22.908239 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:22.908570 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:22.908573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:22.908575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:22.908968 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:22.909111 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:22.909500 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 23:48:22.909801 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":430,"thread_start_us":138,"threads_started":1}
I20260928 23:48:22.913741 11807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:22.914575 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:22.914233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:43968)
0928 23:48:22.914422 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:22.914506 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 23:48:22.914768 11756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
I20260928 23:48:22.915735 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:22.915501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:43984)
0928 23:48:22.915621 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:22.915678 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 23:48:22.915897 11807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:36699: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab.
Entry for principal kudu/127.21.112.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab.
I20260928 23:48:22.921411 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:36699
--webserver_interface=127.21.112.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab
--principal=kudu/127.21.112.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43471,127.21.112.125:36699,127.21.112.124:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:48:22.965265 11807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } attempt: 1
I20260928 23:48:22.966264 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:22.965941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:43994)
0928 23:48:22.966122 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:22.966185 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 23:48:22.967232 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:22.967008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:44000)
0928 23:48:22.967126 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:22.967172 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 23:48:22.967371 11807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:36699: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
I20260928 23:48:23.035565 11807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } attempt: 2
I20260928 23:48:23.036422 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.036121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:44006)
0928 23:48:23.036265 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.036345 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 23:48:23.037227 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.037012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:44018)
0928 23:48:23.037121 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.037169 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:48:23.037400 11807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:36699: Network error: Client connection negotiation failed: client connection to 127.21.112.125:36699: connect: Connection refused (error 111)
W20260928 23:48:23.037242 11814 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:48:23.037647 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:23.037683 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:23.037743 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:23.041923 11814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:23.041991 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:23.042011 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:23.042029 11814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:23.042044 11814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:23.047415 11814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43471,127.21.112.125:36699,127.21.112.124:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab
--principal=kudu/127.21.112.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.112.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:23.048487 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:23.049474 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:23.058074 11819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:23.058074 11820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:23.058089 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:23.058338 11814 server_base.cc:1034] running on GCE node
I20260928 23:48:23.058960 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:23.060106 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:23.061300 11814 hybrid_clock.cc:648] HybridClock initialized: now 1790639303061250 us; error 69 us; skew 500 ppm
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:23.064271 11814 init.cc:377] Logged in from keytab as kudu/127.21.112.125@KRBTEST.COM (short username kudu)
I20260928 23:48:23.065148 11814 webserver.cc:460] Webserver started at http://127.21.112.125:44311/ using document root <none> and password file <none>
I20260928 23:48:23.065836 11814 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:23.065927 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:23.066159 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:23.068193 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data/instance:
uuid: "2a53d7c417c94b669bb11fff5629547c"
format_stamp: "Formatted at 2026-09-28 23:48:23 on dist-test-slave-c3s8"
server_key: "7bb90ab76a42cbc3679177edf3aebedd"
server_key_iv: "2baa4f0f25e3793b3d0ad59c98f75939"
server_key_version: "encryptionkey@0"
I20260928 23:48:23.068758 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal/instance:
uuid: "2a53d7c417c94b669bb11fff5629547c"
format_stamp: "Formatted at 2026-09-28 23:48:23 on dist-test-slave-c3s8"
server_key: "7bb90ab76a42cbc3679177edf3aebedd"
server_key_iv: "2baa4f0f25e3793b3d0ad59c98f75939"
server_key_version: "encryptionkey@0"
I20260928 23:48:23.072489 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 23:48:23.075084 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:23.076336 11814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:48:23.076525 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal
uuid: "2a53d7c417c94b669bb11fff5629547c"
format_stamp: "Formatted at 2026-09-28 23:48:23 on dist-test-slave-c3s8"
server_key: "7bb90ab76a42cbc3679177edf3aebedd"
server_key_iv: "2baa4f0f25e3793b3d0ad59c98f75939"
server_key_version: "encryptionkey@0"
I20260928 23:48:23.076649 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:23.096882 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:23.100647 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:23.100901 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:23.108364 11814 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.125:36699
I20260928 23:48:23.108676 11881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.125:36699 every 8 connection(s)
I20260928 23:48:23.109411 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
I20260928 23:48:23.112462 11882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:23.116504 11882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:23.118539 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11814
I20260928 23:48:23.118675 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-1/wal/instance
I20260928 23:48:23.118983 21953 external_mini_cluster.cc:1428] Setting key 5193209d4068e1e94dbb5dc7d98494f7
WARNING: no policy specified for kudu/127.21.112.124@KRBTEST.COM; defaulting to no policy
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:23.128587 11807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.124@KRBTEST.COM" created.
I20260928 23:48:23.133018 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.118700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.21.112.126:43471)
0928 23:48:23.118868 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.118873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.119792 (+ 919us) server_negotiation.cc:407] Connection header received
0928 23:48:23.120834 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.120841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.120920 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.121041 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.122622 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.123165 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.124371 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.124620 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.124738 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.127436 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.127439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.127442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.127469 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.130486 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.131246 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.131252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.131257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.131353 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.131781 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.131784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.131786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.132023 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.132136 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.132695 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.132826 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:48:23.134184 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.119024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43471 (local address 127.0.0.1:42402)
0928 23:48:23.119609 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.119667 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.120586 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.121247 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.121255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.121825 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.122372 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.122393 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.123309 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.123313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.124220 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.124233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.124795 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.124800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.124976 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.125650 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.125668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.127261 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.130654 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.130664 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.130685 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.131087 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.131458 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.131463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.131465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.131656 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.132136 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.132144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.132432 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.132639 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.132992 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":178,"threads_started":1}
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:48:23.136142 11882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:23.140364 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.129158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:44028)
0928 23:48:23.129302 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.129317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.129405 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.130325 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.130329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.130551 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.130692 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.130699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.131835 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.131841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.132767 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.132779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.133579 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.133583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.133704 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.134217 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.134239 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.135809 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.138347 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.138350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.138352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.138595 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.138940 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.138943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.138945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.139043 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.139753 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.139760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.139895 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.140017 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.140162 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 23:48:23.140949 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.129289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.21.112.125:36699)
0928 23:48:23.129897 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.129910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.129941 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:48:23.129992 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.129998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.130183 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.130354 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.130814 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.131657 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.132902 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.133425 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.133589 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.135940 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.135950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.135964 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.135990 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.138234 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.138726 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.138734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.138736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.138831 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.139159 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.139164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.139167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.139532 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.139693 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.140656 (+ 963us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.140812 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":207,"threads_started":1}
I20260928 23:48:23.143118 11807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:48:23.143931 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.143694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43448)
0928 23:48:23.143818 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.143868 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 23:48:23.144765 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.144451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43450)
0928 23:48:23.144644 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.144696 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 23:48:23.144979 11807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:33367: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
I20260928 23:48:23.147840 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.136746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:44044)
0928 23:48:23.136905 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.136922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.137267 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.137599 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.137602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.137763 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.137829 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.137834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.138766 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.138769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.139510 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.139520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.139998 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.140002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.140086 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.140714 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.140725 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.143671 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.146201 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.146205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.146207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.146426 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.146712 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.146717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.146719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.146799 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.147297 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.147299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.147390 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.147499 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.147653 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 23:48:23.147871 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.137001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.21.112.125:36699)
0928 23:48:23.137353 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.137366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.137377 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:48:23.137427 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.137429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.137461 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.137573 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.137941 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.138638 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.139629 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.139869 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.139989 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.144146 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.144150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.144153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.144180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.146078 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.146537 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.146540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.146541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.146623 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.146884 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.146889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.146891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.147194 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.147298 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.147569 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.147709 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
I20260928 23:48:23.148677 11882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab.
Entry for principal kudu/127.21.112.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab.
I20260928 23:48:23.149353 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.149102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43462)
0928 23:48:23.149212 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.149291 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:48:23.149641 11831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
I20260928 23:48:23.149861 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:33367
--webserver_interface=127.21.112.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab
--principal=kudu/127.21.112.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:43471,127.21.112.125:36699,127.21.112.124:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:48:23.150471 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.150268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43464)
0928 23:48:23.150368 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.150413 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:48:23.150622 11882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:33367: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
I20260928 23:48:23.176169 11807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } attempt: 1
I20260928 23:48:23.177160 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.176808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43470)
0928 23:48:23.177007 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.177075 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 23:48:23.177960 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.177731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43484)
0928 23:48:23.177840 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.177891 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:48:23.178104 11807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:33367: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
I20260928 23:48:23.199906 11882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } attempt: 1
I20260928 23:48:23.200852 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.200545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43494)
0928 23:48:23.200706 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.200772 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 23:48:23.201685 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.201434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:43502)
0928 23:48:23.201574 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.201627 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 23:48:23.201825 11882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:33367: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
I20260928 23:48:23.253299 11807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } attempt: 2
I20260928 23:48:23.254197 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.253899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47832)
0928 23:48:23.254046 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.254117 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 23:48:23.255086 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.254868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47840)
0928 23:48:23.254975 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.255024 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:48:23.255250 11807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:33367: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
W20260928 23:48:23.267133 11890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:48:23.267437 11890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:23.267495 11890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:23.267570 11890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 23:48:23.269999 11882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } attempt: 2
I20260928 23:48:23.270937 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.270591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47850)
0928 23:48:23.270754 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.270839 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 23:48:23.271695 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.271485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47866)
0928 23:48:23.271595 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.271645 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:48:23.271829 11890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:23.271852 11882 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:33367: Network error: Client connection negotiation failed: client connection to 127.21.112.124:33367: connect: Connection refused (error 111)
W20260928 23:48:23.271910 11890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:23.271956 11890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:23.272013 11890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:23.272050 11890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:23.277661 11890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:43471,127.21.112.125:36699,127.21.112.124:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab
--principal=kudu/127.21.112.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.112.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:23.278791 11890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:23.279902 11890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:23.288743 11895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:23.288743 11896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:23.288993 11890 server_base.cc:1034] running on GCE node
W20260928 23:48:23.288754 11898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:23.289739 11890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:23.290889 11890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:23.292122 11890 hybrid_clock.cc:648] HybridClock initialized: now 1790639303292084 us; error 51 us; skew 500 ppm
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:23.295231 11890 init.cc:377] Logged in from keytab as kudu/127.21.112.124@KRBTEST.COM (short username kudu)
I20260928 23:48:23.296329 11890 webserver.cc:460] Webserver started at http://127.21.112.124:41107/ using document root <none> and password file <none>
I20260928 23:48:23.297039 11890 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:23.297303 11890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:23.297675 11890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:23.300047 11890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data/instance:
uuid: "81d9376e8680485493c193c0ddc23dd4"
format_stamp: "Formatted at 2026-09-28 23:48:23 on dist-test-slave-c3s8"
server_key: "71d644459678e1259af2a20c444c7fbb"
server_key_iv: "51580e8ce95dcb531d3f8c2c6eb31a0d"
server_key_version: "encryptionkey@0"
I20260928 23:48:23.300704 11890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal/instance:
uuid: "81d9376e8680485493c193c0ddc23dd4"
format_stamp: "Formatted at 2026-09-28 23:48:23 on dist-test-slave-c3s8"
server_key: "71d644459678e1259af2a20c444c7fbb"
server_key_iv: "51580e8ce95dcb531d3f8c2c6eb31a0d"
server_key_version: "encryptionkey@0"
I20260928 23:48:23.306327 11890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 23:48:23.309322 11905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:23.310909 11890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:23.311091 11890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal
uuid: "81d9376e8680485493c193c0ddc23dd4"
format_stamp: "Formatted at 2026-09-28 23:48:23 on dist-test-slave-c3s8"
server_key: "71d644459678e1259af2a20c444c7fbb"
server_key_iv: "51580e8ce95dcb531d3f8c2c6eb31a0d"
server_key_version: "encryptionkey@0"
I20260928 23:48:23.311224 11890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:23.324767 11890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:23.328334 11890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:23.328646 11890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:23.336431 11890 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.124:33367
I20260928 23:48:23.336810 11957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.124:33367 every 8 connection(s)
I20260928 23:48:23.337523 11890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
I20260928 23:48:23.340538 11958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:23.344322 11958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:23.347034 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 11890
I20260928 23:48:23.347168 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790639014951155-21953-0/minicluster-data/master-2/wal/instance
I20260928 23:48:23.347445 21953 external_mini_cluster.cc:1428] Setting key 5bfc6e6fbc52cb0fb0d888266e665591
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:48:23.354408 11807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } attempt: 3
I20260928 23:48:23.358800 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.346255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.21.112.126:43471)
0928 23:48:23.346465 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.346469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.347353 (+ 884us) server_negotiation.cc:407] Connection header received
0928 23:48:23.348865 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.348869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.348936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.349035 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.350007 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.350524 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.351533 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.351756 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.351885 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.354564 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.354568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.354571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.354602 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.356449 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.356947 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.356953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.356954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.357063 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.357472 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.357476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.357478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.357717 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.357787 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.358468 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.358599 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:48:23.360248 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.348151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.21.112.124:33367)
0928 23:48:23.348548 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.348558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.348596 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:48:23.348719 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.348735 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.348915 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.349110 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.349993 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.350850 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.351715 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.352090 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.352209 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.355295 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.355302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.355316 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.355341 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.357747 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.358267 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.358272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.358274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.358363 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.358675 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.358679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.358680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.359004 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.359186 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.359617 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.359750 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":156,"threads_started":1}
I20260928 23:48:23.360248 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.346565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:43471 (local address 127.0.0.1:57136)
0928 23:48:23.347144 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.347205 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.348307 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.349090 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.349096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.349586 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.349861 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.349875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.350643 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.350650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.351405 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.351414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.351919 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.351923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.352087 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.352703 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.352726 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.354388 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.356548 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.356559 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.356569 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.356839 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.357167 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.357170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.357172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.357360 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.357870 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.357877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.358160 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.358964 (+ 804us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.359261 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":174,"threads_started":1}
I20260928 23:48:23.361653 11958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:23.363157 11882 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } attempt: 3
I20260928 23:48:23.366050 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.354933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47888)
0928 23:48:23.355069 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.355085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.355156 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.355978 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.355983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.356146 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.356266 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.356275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.357064 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.357068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.357871 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.357880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.358470 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.358474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.358567 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.359259 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.359273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.360679 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.364469 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.364475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.364481 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.364738 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.365050 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.365053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.365055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.365139 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.365571 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.365574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.365669 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.365755 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.365880 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 23:48:23.366707 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.354991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.21.112.124:33367)
0928 23:48:23.355302 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.355311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.355322 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:48:23.355350 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.355353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.355393 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.355484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.356405 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.356964 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.357980 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.358348 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.358465 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.361831 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.361835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.361838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.361863 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.364342 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.364857 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.364860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.364862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.364950 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.365238 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.365241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.365242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.365436 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.366302 (+ 866us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.366388 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.366508 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:23.370488 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15: Bootstrap starting.
I20260928 23:48:23.373581 11807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:23.374086 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.363542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47902)
0928 23:48:23.363706 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.363720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.363791 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.364075 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.364081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.364270 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.364376 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.364387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.365432 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.365436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.366258 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.366272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.367285 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.367288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.367374 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.367831 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.367845 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.369207 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.371232 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.371238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.371244 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.371524 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.372904 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.372909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.372911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.373013 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.373468 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.373470 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.373596 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.373754 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.373909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 23:48:23.374264 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.363337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.21.112.126:43471)
0928 23:48:23.363449 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.363454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.363470 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:48:23.363703 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.363706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.363758 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.363810 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.364715 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.365443 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.366558 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.366895 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.366962 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.370750 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.370753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.370756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.370780 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.372275 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.372764 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.372768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.372770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.372850 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.373159 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.373163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.373164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.373368 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.373529 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.374014 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.374132 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:48:23.374581 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.362319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.21.112.125:36699)
0928 23:48:23.362556 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.362560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.362733 (+ 173us) server_negotiation.cc:407] Connection header received
0928 23:48:23.362916 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.362920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.363003 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.363063 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.364548 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.365361 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.366529 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.366883 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.366962 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.370593 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.370597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.370599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.370627 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.372689 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.373254 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.373258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.373259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.373333 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.373677 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.373680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.373682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.373868 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.373933 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.374279 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.374439 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 23:48:23.374603 11807 log.cc:824] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:23.374781 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.363587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.21.112.124:33367)
0928 23:48:23.363695 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.363701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.363763 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:48:23.363876 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.363879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.363923 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.364501 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.364593 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.365315 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.366626 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.367159 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.367286 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.369344 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.369348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.369350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.369383 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.371098 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.371918 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.371923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.371928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.372807 (+ 879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.373127 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.373130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.373132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.373358 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.374391 (+ 1033us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.374464 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.374564 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":1792}
I20260928 23:48:23.374963 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.362212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:36699 (local address 127.0.0.1:55474)
0928 23:48:23.362663 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.362677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.362797 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.363948 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.363951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.364265 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.364397 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.364405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.365514 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.365520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.366382 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.366392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.367474 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.367479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.367611 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.368040 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.368055 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.370429 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.372921 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.372923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.372925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.373152 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.373463 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.373465 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.373467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.373571 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.374005 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.374008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.374090 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.374706 (+ 616us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.374826 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388}
I20260928 23:48:23.376889 11807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15: No bootstrap required, opened a new log
I20260928 23:48:23.377800 11958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:23.378074 11882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: Bootstrap starting.
I20260928 23:48:23.380534 11807 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.380760 11807 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03946032a9540ebada12617b3ac9e15, State: Initialized, Role: FOLLOWER
I20260928 23:48:23.380893 11882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:23.381134 11807 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.381655 11965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.381795 11882 log.cc:824] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:23.381870 11965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.382620 11807 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:48:23.384299 11882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: No bootstrap required, opened a new log
W20260928 23:48:23.388513 11977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 23:48:23.388414 11882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.388654 11882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a53d7c417c94b669bb11fff5629547c, State: Initialized, Role: FOLLOWER
I20260928 23:48:23.388679 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.377689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.21.112.125:36699)
0928 23:48:23.377814 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.377818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.377831 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:48:23.377875 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.377878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.377920 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.377997 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.378814 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.379549 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.380885 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.381211 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.381288 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.384578 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.384582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.384585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.384610 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.386422 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.386925 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.386929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.386931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.387025 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.387340 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.387343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.387345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.387568 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.387759 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.388399 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.388519 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:48:23.388700 11977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:23.389021 11882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.389513 11979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.389734 11979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.390965 11882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:23.387859 11739 master_runner.cc:186] Error getting master registration for 127.21.112.125:36699: OK, instance_id { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" instance_seqno: 1790639303106344 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:48:23.391675 11739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:48:23.394112 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 23:48:23 dist-test-slave-c3s8 krb5kdc[11726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639303, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:48:23.395480 11781 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d03946032a9540ebada12617b3ac9e15"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57152
I20260928 23:48:23.395730 11781 raft_consensus.cc:482] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:48:23.395861 11781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:23.397115 11781 raft_consensus.cc:504] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.398249 11781 leader_election.cc:288] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [CANDIDATE]: Term 1 election: Requested vote from peers 2a53d7c417c94b669bb11fff5629547c (127.21.112.125:36699), 81d9376e8680485493c193c0ddc23dd4 (127.21.112.124:33367)
I20260928 23:48:23.398795 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.378282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:33367 (local address 127.0.0.1:47914)
0928 23:48:23.382321 (+ 4039us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:23.382337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:23.382470 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:23.389668 (+ 7198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:23.389673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:23.389874 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:23.390013 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.390020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.391099 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.391103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.392020 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:23.392032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.392776 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.392779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:23.392855 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.393380 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:23.393395 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:23.395111 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:23.397063 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.397067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.397069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.397314 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.397650 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:23.397652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:23.397654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:23.397735 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:23.398256 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:23.398262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:23.398363 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 23:48:23.398458 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:48:23.398610 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3952}
I20260928 23:48:23.399259 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.378496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.21.112.124:33367)
0928 23:48:23.378748 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.378753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.382404 (+ 3651us) server_negotiation.cc:407] Connection header received
0928 23:48:23.382578 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.382582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.382623 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.382673 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.390155 (+ 7482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.390930 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.392150 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.392533 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.392619 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.395249 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.395252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.395254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.395277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.396936 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.397444 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.397447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.397449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.397528 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.397815 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.397818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.397819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.398127 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.398221 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.399012 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.399129 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260928 23:48:23.400815 11856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d03946032a9540ebada12617b3ac9e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2a53d7c417c94b669bb11fff5629547c"
I20260928 23:48:23.401080 11856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:23.402503 11958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: Bootstrap starting.
I20260928 23:48:23.402552 11856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03946032a9540ebada12617b3ac9e15 in term 1.
I20260928 23:48:23.401923 11814 master_runner.cc:186] Error getting master registration for 127.21.112.124:33367: OK, instance_id { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" instance_seqno: 1790639303334245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:48:23.405033 11814 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:48:23.405270 11756 leader_election.cc:302] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a53d7c417c94b669bb11fff5629547c, d03946032a9540ebada12617b3ac9e15; no voters:
I20260928 23:48:23.405624 11958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:23.406116 11965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:23.406337 11965 raft_consensus.cc:686] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [term 1 LEADER]: Becoming Leader. State: Replica: d03946032a9540ebada12617b3ac9e15, State: Running, Role: LEADER
I20260928 23:48:23.406570 11958 log.cc:824] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:23.406697 11965 consensus_queue.cc:237] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.408737 11965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03946032a9540ebada12617b3ac9e15. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.408878 11965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:23.409017 11958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: No bootstrap required, opened a new log
I20260928 23:48:23.414361 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.403304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.21.112.126:43471)
0928 23:48:23.403448 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.403453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.404132 (+ 679us) server_negotiation.cc:407] Connection header received
0928 23:48:23.404429 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.404432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.404484 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.404538 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.405674 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.406421 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.407559 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.407856 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.407931 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.409917 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.409921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.409924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.409965 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.411916 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.412548 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.412552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.412554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.412672 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.412986 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.412989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.412991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.413208 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.413317 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.413757 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.413879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:48:23.415154 11958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
I20260928 23:48:23.415457 11958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d9376e8680485493c193c0ddc23dd4, State: Initialized, Role: FOLLOWER
I20260928 23:48:23.415841 11958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } }
W20260928 23:48:23.416618 11990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:23.416697 11990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:23.418483 11962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.406056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.21.112.124:33367)
0928 23:48:23.406162 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.406165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.406179 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:48:23.406213 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.406216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.406253 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.406330 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.408038 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.408736 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.410636 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.410972 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.411051 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.413720 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.413724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.413727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.413754 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.415566 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.416594 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.416598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.416600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.416689 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.417252 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.417257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.417261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.417488 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.417645 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.418197 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.418325 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:48:23.421384 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:23.403667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.21.112.125:36699)
0928 23:48:23.405674 (+ 2007us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.405678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.405690 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:48:23.405724 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.405727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.405787 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.405857 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.409917 (+ 4060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.410727 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.413646 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:23.413996 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:23.414085 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:23.417643 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:23.417647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:23.417650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:23.417679 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:23.419430 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.419936 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.419941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.419943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.420033 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:23.420342 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:23.420346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:23.420347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:23.420574 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:23.420709 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:23.421097 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 23:48:23.421222 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911}
I20260928 23:48:23.423069 11958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:23.423116 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d03946032a9540ebada12617b3ac9e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "81d9376e8680485493c193c0ddc23dd4"
I20260928 23:48:23.423352 11932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:23.425411 11932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03946032a9540ebada12617b3ac9e15 in term 1.
I20260928 23:48:23.427067 12001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:23.427985 12001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:23.430574 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.431571 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.436417 11932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [term 1 FOLLOWER]: Refusing update from remote peer d03946032a9540ebada12617b3ac9e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:48:23.437143 11998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:48:23.437719 11856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [term 1 FOLLOWER]: Refusing update from remote peer d03946032a9540ebada12617b3ac9e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:48:23.438271 11965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:48:23.443198 12015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:23.443295 12015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:23.447154 11979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03946032a9540ebada12617b3ac9e15. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.447270 11979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.448760 11965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.448894 11965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:23.450423 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d03946032a9540ebada12617b3ac9e15. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.450552 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.450644 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.450718 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.451745 11965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.451866 11965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:23.453105 11979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d03946032a9540ebada12617b3ac9e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03946032a9540ebada12617b3ac9e15" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 43471 } } peers { permanent_uuid: "2a53d7c417c94b669bb11fff5629547c" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 36699 } } peers { permanent_uuid: "81d9376e8680485493c193c0ddc23dd4" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 33367 } } }
I20260928 23:48:23.453228 11979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:23.457643 12019 mvcc.cc:205] Tried to move back new op lower bound from 7334458586868162560 to 7334458586757869568. Current Snapshot: MvccSnapshot[applied={T|T < 7334458586868162560}]
I20260928 23:48:23.459951 12001 catalog_manager.cc:1347] Generated new cluster ID: b7cf6e7b836d463d93100f6a8161a41a
I20260928 23:48:23.460028 12001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:23.480926 12001 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:23.482357 12001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:23.495039 12001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d03946032a9540ebada12617b3ac9e15: Generated new TSK 0
I20260928 23:48:23.495903 12001 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:48:23.532495 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.529103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.21.112.125:36699)
0928 23:48:23.529241 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.529246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.531313 (+ 2067us) server_negotiation.cc:407] Connection header received
0928 23:48:23.531543 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.531547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.531601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.531657 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.532333 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55504: BlockingRecv error: recv got EOF from 127.0.0.1:55504 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 23:48:23.534013 11962 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.529986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.21.112.124:33367)
0928 23:48:23.530134 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.530139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.533021 (+ 2882us) server_negotiation.cc:407] Connection header received
0928 23:48:23.533198 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.533201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.533251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.533303 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.533878 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47936: BlockingRecv error: recv got EOF from 127.0.0.1:47936 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 23:48:23.535457 11811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.530763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.21.112.126:43471)
0928 23:48:23.530908 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.530912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.534550 (+ 3638us) server_negotiation.cc:407] Connection header received
0928 23:48:23.534718 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.534722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.534773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.534822 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.535324 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57176: BlockingRecv error: recv got EOF from 127.0.0.1:57176 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 23:48:23.539760 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.538063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.21.112.125:36699)
0928 23:48:23.538180 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.538186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.538719 (+ 533us) server_negotiation.cc:407] Connection header received
0928 23:48:23.538948 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.538951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.539028 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.539082 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.539648 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55516: BlockingRecv error: recv got EOF from 127.0.0.1:55516 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 23:48:23.545305 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.543729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.21.112.125:36699)
0928 23:48:23.543837 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.543841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.544326 (+ 485us) server_negotiation.cc:407] Connection header received
0928 23:48:23.544536 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.544539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.544598 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.544650 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.545188 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55522: BlockingRecv error: recv got EOF from 127.0.0.1:55522 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 23:48:23.551796 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.550046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.21.112.125:36699)
0928 23:48:23.550166 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.550171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.550762 (+ 591us) server_negotiation.cc:407] Connection header received
0928 23:48:23.550986 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.550990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.551048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.551105 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.551670 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55530: BlockingRecv error: recv got EOF from 127.0.0.1:55530 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 23:48:23.560405 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.558511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.21.112.125:36699)
0928 23:48:23.558701 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.558706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.559313 (+ 607us) server_negotiation.cc:407] Connection header received
0928 23:48:23.559569 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.559573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.559635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.559692 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.560287 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55540: BlockingRecv error: recv got EOF from 127.0.0.1:55540 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260928 23:48:23.573310 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.571396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.21.112.125:36699)
0928 23:48:23.571544 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.571550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.572179 (+ 629us) server_negotiation.cc:407] Connection header received
0928 23:48:23.572420 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.572425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.572486 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.572561 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.573182 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55552: BlockingRecv error: recv got EOF from 127.0.0.1:55552 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 23:48:23.594044 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.592189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.21.112.125:36699)
0928 23:48:23.592318 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.592322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.592948 (+ 626us) server_negotiation.cc:407] Connection header received
0928 23:48:23.593161 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.593166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.593223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.593275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.593923 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55568: BlockingRecv error: recv got EOF from 127.0.0.1:55568 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:48:23.630275 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.628785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.21.112.125:36699)
0928 23:48:23.628922 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.628927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.629252 (+ 325us) server_negotiation.cc:407] Connection header received
0928 23:48:23.629485 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.629489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.629602 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.629703 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.630154 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55572: BlockingRecv error: recv got EOF from 127.0.0.1:55572 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 23:48:23.698182 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.696842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.21.112.125:36699)
0928 23:48:23.697009 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.697014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.697164 (+ 150us) server_negotiation.cc:407] Connection header received
0928 23:48:23.697352 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.697355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.697418 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.697513 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.698065 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55588: BlockingRecv error: recv got EOF from 127.0.0.1:55588 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 23:48:23.829784 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:23.828582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.21.112.125:36699)
0928 23:48:23.828736 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:23.828740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:23.828815 (+ 75us) server_negotiation.cc:407] Connection header received
0928 23:48:23.828944 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:23.828947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:23.828995 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:23.829082 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:23.829660 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55590: BlockingRecv error: recv got EOF from 127.0.0.1:55590 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 23:48:24.089035 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:24.087906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.21.112.125:36699)
0928 23:48:24.088069 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:24.088073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:24.088133 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:48:24.088259 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:24.088261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:24.088311 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:24.088404 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:24.088930 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55606: BlockingRecv error: recv got EOF from 127.0.0.1:55606 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:48:24.418099 11990 catalog_manager.cc:1259] Loaded cluster ID: b7cf6e7b836d463d93100f6a8161a41a
I20260928 23:48:24.418185 11990 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: loading cluster ID for follower catalog manager: success
I20260928 23:48:24.419911 11990 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: acquiring CA information for follower catalog manager: success
I20260928 23:48:24.421113 11990 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a53d7c417c94b669bb11fff5629547c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:48:24.444691 12015 catalog_manager.cc:1259] Loaded cluster ID: b7cf6e7b836d463d93100f6a8161a41a
I20260928 23:48:24.444780 12015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: loading cluster ID for follower catalog manager: success
I20260928 23:48:24.446684 12015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: acquiring CA information for follower catalog manager: success
I20260928 23:48:24.447952 12015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81d9376e8680485493c193c0ddc23dd4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:48:24.607846 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:24.603442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.21.112.125:36699)
0928 23:48:24.603731 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:24.603735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:24.603753 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:48:24.603848 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:24.603851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:24.603979 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:24.604092 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:24.604350 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.605065 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.606207 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.606438 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.606577 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:24.606876 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:24.607336 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:24.607435 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:24.607657 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 23:48:24.607715 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":138,"threads_started":1}
I20260928 23:48:24.616111 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:24.611623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.21.112.124:33367)
0928 23:48:24.611861 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:24.611865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:24.611909 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:48:24.612100 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:24.612103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:24.612236 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:24.612345 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:24.612675 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.613234 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.614294 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.614555 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.614710 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:24.615078 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:24.615557 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:24.615659 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:24.615820 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 23:48:24.615888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260928 23:48:24.623564 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:24.620099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.21.112.124:33367)
0928 23:48:24.620209 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:24.620212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:24.620222 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:48:24.620258 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:24.620260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:24.620296 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:24.620353 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:24.620657 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.621439 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.622364 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.622651 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.622761 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:24.623040 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:24.623122 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:24.623205 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:24.623352 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 23:48:24.623412 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:48:24.624183 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:24.619551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.21.112.125:36699)
0928 23:48:24.619703 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:24.619708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:24.619948 (+ 240us) server_negotiation.cc:407] Connection header received
0928 23:48:24.620490 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:24.620496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:24.620558 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:24.620611 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:24.620948 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.621461 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.622857 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.623144 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.623263 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:24.623664 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:24.623748 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:24.623792 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:24.623981 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:48:24.624048 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 23:48:24.624836 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:24.619410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.21.112.126:43471)
0928 23:48:24.619675 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:24.619680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:24.620007 (+ 327us) server_negotiation.cc:407] Connection header received
0928 23:48:24.620130 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:24.620134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:24.620275 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:24.620407 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:24.620743 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.621559 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.622902 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:24.623287 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:24.623399 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:24.623802 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:24.624381 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:24.624463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:24.624612 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 23:48:24.624684 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260928 23:48:24.628306 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11739
I20260928 23:48:24.634213 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11814
I20260928 23:48:24.641108 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 11890
2026-09-28T23:48:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3135 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:48:24 dist-test-slave-c3s8 krb5kdc[12107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:48:24 dist-test-slave-c3s8 krb5kdc[12107](info): set up 2 sockets
Sep 28 23:48:24 dist-test-slave-c3s8 krb5kdc[12107](info): commencing operation
krb5kdc: starting...
W20260928 23:48:25.702353 21953 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/test-user.keytab.
Sep 28 23:48:25 dist-test-slave-c3s8 krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.112.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
Entry for principal kudu/127.21.112.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab.
I20260928 23:48:25.832659 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:45817
--webserver_interface=127.21.112.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:45817,127.21.112.125:46679,127.21.112.124:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:48:25.946414 12120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:48:25.946704 12120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:25.946728 12120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:25.946822 12120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:25.950958 12120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:25.951023 12120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:25.951042 12120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:25.951061 12120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:25.951076 12120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:25.956406 12120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:45817,127.21.112.125:46679,127.21.112.124:39999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.126.keytab
--principal=kudu/127.21.112.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.126:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.112.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:25.957427 12120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:25.958588 12120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:25.967271 12126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:25.967278 12125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:25.967320 12120 server_base.cc:1034] running on GCE node
W20260928 23:48:25.967269 12128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:25.968032 12120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:25.969200 12120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:25.970392 12120 hybrid_clock.cc:648] HybridClock initialized: now 1790639305970361 us; error 43 us; skew 500 ppm
Sep 28 23:48:25 dist-test-slave-c3s8 krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:25.973295 12120 init.cc:377] Logged in from keytab as kudu/127.21.112.126@KRBTEST.COM (short username kudu)
I20260928 23:48:25.974243 12120 webserver.cc:460] Webserver started at http://127.21.112.126:38843/ using document root <none> and password file <none>
I20260928 23:48:25.974833 12120 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:25.974929 12120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:25.975162 12120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:25.977296 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data/instance:
uuid: "fda25b336dc84acbbee7b0cc15b50d98"
format_stamp: "Formatted at 2026-09-28 23:48:25 on dist-test-slave-c3s8"
server_key: "0b70a2dda4cf4bd94982b9a15e6f7584"
server_key_iv: "41cabd80702b9a61895815384a8e1cdb"
server_key_version: "encryptionkey@0"
I20260928 23:48:25.977914 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal/instance:
uuid: "fda25b336dc84acbbee7b0cc15b50d98"
format_stamp: "Formatted at 2026-09-28 23:48:25 on dist-test-slave-c3s8"
server_key: "0b70a2dda4cf4bd94982b9a15e6f7584"
server_key_iv: "41cabd80702b9a61895815384a8e1cdb"
server_key_version: "encryptionkey@0"
I20260928 23:48:25.981810 12120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 23:48:25.984417 12135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:25.985921 12120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 23:48:25.986068 12120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal
uuid: "fda25b336dc84acbbee7b0cc15b50d98"
format_stamp: "Formatted at 2026-09-28 23:48:25 on dist-test-slave-c3s8"
server_key: "0b70a2dda4cf4bd94982b9a15e6f7584"
server_key_iv: "41cabd80702b9a61895815384a8e1cdb"
server_key_version: "encryptionkey@0"
I20260928 23:48:25.986192 12120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:26.006989 12120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:26.011098 12120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:26.011343 12120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:26.022014 12120 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.126:45817
I20260928 23:48:26.022315 12187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.126:45817 every 8 connection(s)
I20260928 23:48:26.023283 12120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/data/info.pb
I20260928 23:48:26.026469 12188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:26.029474 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 12120
I20260928 23:48:26.029634 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-0/wal/instance
I20260928 23:48:26.029985 21953 external_mini_cluster.cc:1428] Setting key 215a88f78ee561f363a8938b74455fae
I20260928 23:48:26.031522 12188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.112.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.125@KRBTEST.COM" created.
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:26.049376 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.034279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:45817 (local address 127.0.0.1:46888)
0928 23:48:26.034855 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.034915 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.035980 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.036913 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.036918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.037370 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.037774 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.037792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.039373 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.039375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.040125 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.040135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.040942 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.040946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.041088 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.041846 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.041874 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.043696 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.046089 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.046098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.046107 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.046373 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.046789 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.046792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.046793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.046931 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.047811 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.047817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.047969 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.048144 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.048468 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"thread_start_us":172,"threads_started":1}
I20260928 23:48:26.049379 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.035300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.21.112.126:45817)
0928 23:48:26.035681 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.035695 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.035750 (+ 55us) server_negotiation.cc:407] Connection header received
0928 23:48:26.036344 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.036350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.036650 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.036887 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.037955 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.039219 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.040251 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.040719 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.040902 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.043842 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.043850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.043867 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.043898 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.045965 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.046504 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.046511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.046513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.046681 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.047038 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.047042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.047043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.047656 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.047818 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.048270 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.048564 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260928 23:48:26.052243 12188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:26.053094 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.052802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34324)
0928 23:48:26.052949 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.053028 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 23:48:26.053315 12137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
I20260928 23:48:26.054293 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.054074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34332)
0928 23:48:26.054192 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.054239 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 23:48:26.054479 12188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:46679: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab.
Entry for principal kudu/127.21.112.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab.
I20260928 23:48:26.062248 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:46679
--webserver_interface=127.21.112.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab
--principal=kudu/127.21.112.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:45817,127.21.112.125:46679,127.21.112.124:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:48:26.103777 12188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } attempt: 1
I20260928 23:48:26.104805 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.104456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34334)
0928 23:48:26.104653 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.104721 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 23:48:26.105676 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.105404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34338)
0928 23:48:26.105524 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.105609 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 23:48:26.105814 12188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:46679: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
I20260928 23:48:26.173974 12188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } attempt: 2
I20260928 23:48:26.174901 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.174610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34346)
0928 23:48:26.174752 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.174823 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 23:48:26.175626 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.175403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34362)
0928 23:48:26.175517 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.175575 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 23:48:26.175772 12188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.125:46679: Network error: Client connection negotiation failed: client connection to 127.21.112.125:46679: connect: Connection refused (error 111)
W20260928 23:48:26.176748 12195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:48:26.177032 12195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:26.177084 12195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:26.177174 12195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:26.181217 12195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:26.181270 12195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:26.181290 12195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:26.181306 12195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:26.181365 12195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:26.186786 12195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:45817,127.21.112.125:46679,127.21.112.124:39999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.125.keytab
--principal=kudu/127.21.112.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.125:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.112.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:26.187870 12195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:26.188920 12195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:26.197822 12201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:26.197839 12200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:26.197815 12203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:26.198086 12195 server_base.cc:1034] running on GCE node
I20260928 23:48:26.198748 12195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:26.199910 12195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:26.201102 12195 hybrid_clock.cc:648] HybridClock initialized: now 1790639306201086 us; error 27 us; skew 500 ppm
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:26.203914 12195 init.cc:377] Logged in from keytab as kudu/127.21.112.125@KRBTEST.COM (short username kudu)
I20260928 23:48:26.204768 12195 webserver.cc:460] Webserver started at http://127.21.112.125:33687/ using document root <none> and password file <none>
I20260928 23:48:26.205295 12195 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:26.205353 12195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:26.205513 12195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:26.207572 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data/instance:
uuid: "080c0bdba2f64b20b77bf110de744445"
format_stamp: "Formatted at 2026-09-28 23:48:26 on dist-test-slave-c3s8"
server_key: "102059aec3c563e2ef064796ee6e1a72"
server_key_iv: "426d618f6be6d5b9f145fd8518f6984e"
server_key_version: "encryptionkey@0"
I20260928 23:48:26.208107 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal/instance:
uuid: "080c0bdba2f64b20b77bf110de744445"
format_stamp: "Formatted at 2026-09-28 23:48:26 on dist-test-slave-c3s8"
server_key: "102059aec3c563e2ef064796ee6e1a72"
server_key_iv: "426d618f6be6d5b9f145fd8518f6984e"
server_key_version: "encryptionkey@0"
I20260928 23:48:26.211918 12195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 23:48:26.214460 12210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:26.215724 12195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:48:26.215850 12195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal
uuid: "080c0bdba2f64b20b77bf110de744445"
format_stamp: "Formatted at 2026-09-28 23:48:26 on dist-test-slave-c3s8"
server_key: "102059aec3c563e2ef064796ee6e1a72"
server_key_iv: "426d618f6be6d5b9f145fd8518f6984e"
server_key_version: "encryptionkey@0"
I20260928 23:48:26.215962 12195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:26.234719 12195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:26.237916 12195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:26.238131 12195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:26.245680 12195 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.125:46679
I20260928 23:48:26.245996 12262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.125:46679 every 8 connection(s)
I20260928 23:48:26.246752 12195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/data/info.pb
I20260928 23:48:26.248870 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 12195
I20260928 23:48:26.248994 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-1/wal/instance
I20260928 23:48:26.249301 21953 external_mini_cluster.cc:1428] Setting key 3a0a7384e9ef49c8c52c6dbcc4443058
I20260928 23:48:26.250000 12263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:26.254070 12263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.112.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.112.124@KRBTEST.COM" created.
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:26.266978 12188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } attempt: 3
I20260928 23:48:26.269034 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.256053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.21.112.126:45817)
0928 23:48:26.256196 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.256200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.257242 (+ 1042us) server_negotiation.cc:407] Connection header received
0928 23:48:26.258262 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.258265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.258328 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.258433 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.259589 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.260120 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.261793 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.262059 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.262198 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.264952 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.264960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.264962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.264987 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.266567 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.267143 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.267147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.267149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.267262 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.267687 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.267694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.267698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.267942 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.268045 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.268744 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.268872 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:48:26.270032 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.256567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:45817 (local address 127.0.0.1:46904)
0928 23:48:26.257086 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.257128 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.258055 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.258606 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.258612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.258947 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.259397 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.259409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.260250 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.260254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.261660 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.261675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.262239 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.262244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.262432 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.263057 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.263073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.264773 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.266689 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.266695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.266711 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.267013 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.267378 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.267381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.267383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.267567 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.268118 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.268124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.268436 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.268686 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.269024 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260928 23:48:26.271489 12263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.112.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab.
Entry for principal kudu/127.21.112.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab.
I20260928 23:48:26.277150 21953 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzqclLI/build/debug/bin/kudu
/tmp/dist-test-taskzqclLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:39999
--webserver_interface=127.21.112.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab
--principal=kudu/127.21.112.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.112.84:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.112.126:45817,127.21.112.125:46679,127.21.112.124:39999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:48:26.278188 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.267443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34370)
0928 23:48:26.267655 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.267671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.267736 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.268453 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.268458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.268637 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.269246 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.269260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.270461 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.270467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.271149 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.271157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.271969 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.271973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.272089 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.272719 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.272735 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.274225 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.276486 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.276489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.276491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.276694 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.277003 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.277007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.277009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.277098 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.277649 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.277652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.277752 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.277879 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.278021 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 23:48:26.278276 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.267602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.21.112.125:46679)
0928 23:48:26.267981 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.268001 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.268027 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:48:26.268071 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.268079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.268289 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.268425 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.269363 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.270349 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.271261 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.271842 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.271972 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.274366 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.274375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.274389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.274413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.276361 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.276806 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.276812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.276814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.276907 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.277192 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.277197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.277199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.277515 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.277686 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.277974 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.278111 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":91,"threads_started":1}
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:48:26.280820 12188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:26.281661 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.281350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35500)
0928 23:48:26.281496 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.281577 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 23:48:26.282332 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.282085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35510)
0928 23:48:26.282224 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.282271 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 23:48:26.282490 12188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:39999: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
I20260928 23:48:26.283501 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.272015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34386)
0928 23:48:26.272113 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.272126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.272204 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.272767 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.272769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.272946 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.273053 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.273058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.274128 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.274131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.274935 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.274947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.275694 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.275697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.275860 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.276363 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.276382 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.278914 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.281311 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.281314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.281316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.281599 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.282486 (+ 887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.282489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.282491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.282585 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.282991 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.282995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.283083 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.283183 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.283327 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":51}
I20260928 23:48:26.283547 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.272162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.21.112.125:46679)
0928 23:48:26.272524 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.272536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.272551 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:48:26.272582 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.272585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.272629 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.272731 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.273167 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.273997 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.275107 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.275574 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.275697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.279454 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.279457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.279459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.279479 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.281188 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.281963 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.281968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.281972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.282070 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.282698 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.282703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.282705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.282886 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.282991 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.283238 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.283383 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":275,"thread_start_us":138,"threads_started":1}
I20260928 23:48:26.284164 12263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:26.284838 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.284596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35524)
0928 23:48:26.284702 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.284774 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 23:48:26.285038 12212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
I20260928 23:48:26.285889 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.285681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35532)
0928 23:48:26.285787 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.285831 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:48:26.286044 12263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:39999: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
I20260928 23:48:26.313617 12188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } attempt: 1
I20260928 23:48:26.314535 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.314197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35548)
0928 23:48:26.314382 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.314446 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 23:48:26.315302 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.315085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35554)
0928 23:48:26.315195 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.315243 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:48:26.315430 12188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:39999: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
I20260928 23:48:26.335290 12263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } attempt: 1
I20260928 23:48:26.336263 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.335940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35558)
0928 23:48:26.336115 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.336180 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 23:48:26.337033 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.336802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35566)
0928 23:48:26.336913 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.336968 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 23:48:26.337168 12263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:39999: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
I20260928 23:48:26.390614 12188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } attempt: 2
I20260928 23:48:26.391657 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.391199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35574)
0928 23:48:26.391508 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.391573 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 23:48:26.392540 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.392293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35576)
0928 23:48:26.392412 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.392467 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 23:48:26.392678 12188 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:39999: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
W20260928 23:48:26.396827 12270 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:48:26.397147 12270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:48:26.397208 12270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:48:26.397289 12270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:48:26.401377 12270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:48:26.401571 12270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:48:26.401624 12270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:48:26.401661 12270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:48:26.401695 12270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:48:26.405345 12263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } attempt: 2
I20260928 23:48:26.406208 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.405922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35592)
0928 23:48:26.406062 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.406127 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 23:48:26.407009 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.406792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35598)
0928 23:48:26.406905 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.406951 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 23:48:26.407123 12263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.112.124:39999: Network error: Client connection negotiation failed: client connection to 127.21.112.124:39999: connect: Connection refused (error 111)
I20260928 23:48:26.407517 12270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.112.84:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.112.126:45817,127.21.112.125:46679,127.21.112.124:39999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/krb5kdc/kudu_127.21.112.124.keytab
--principal=kudu/127.21.112.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.112.124:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.112.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 22:58:03 UTC on ccycloud.quasar-dycpvp.root.comops.site
build id 4678930
I20260928 23:48:26.408639 12270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:48:26.409780 12270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:48:26.418617 12279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:26.418622 12277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:48:26.418622 12276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:48:26.418843 12270 server_base.cc:1034] running on GCE node
I20260928 23:48:26.419473 12270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:48:26.420660 12270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:48:26.421816 12270 hybrid_clock.cc:648] HybridClock initialized: now 1790639306421779 us; error 48 us; skew 500 ppm
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:48:26.424556 12270 init.cc:377] Logged in from keytab as kudu/127.21.112.124@KRBTEST.COM (short username kudu)
I20260928 23:48:26.425452 12270 webserver.cc:460] Webserver started at http://127.21.112.124:41789/ using document root <none> and password file <none>
I20260928 23:48:26.426103 12270 fs_manager.cc:359] Metadata directory not provided
I20260928 23:48:26.426200 12270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:48:26.426435 12270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:48:26.428494 12270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data/instance:
uuid: "c60fca428b9f46fc862c6be732941039"
format_stamp: "Formatted at 2026-09-28 23:48:26 on dist-test-slave-c3s8"
server_key: "dc2a0741d5f82c6af0d0104c49a065d7"
server_key_iv: "4bdbcfda2d57922d06c110d9fd77470e"
server_key_version: "encryptionkey@0"
I20260928 23:48:26.429090 12270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal/instance:
uuid: "c60fca428b9f46fc862c6be732941039"
format_stamp: "Formatted at 2026-09-28 23:48:26 on dist-test-slave-c3s8"
server_key: "dc2a0741d5f82c6af0d0104c49a065d7"
server_key_iv: "4bdbcfda2d57922d06c110d9fd77470e"
server_key_version: "encryptionkey@0"
I20260928 23:48:26.433430 12270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 23:48:26.436165 12286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:48:26.437505 12270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:48:26.437721 12270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data,/tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal
uuid: "c60fca428b9f46fc862c6be732941039"
format_stamp: "Formatted at 2026-09-28 23:48:26 on dist-test-slave-c3s8"
server_key: "dc2a0741d5f82c6af0d0104c49a065d7"
server_key_iv: "4bdbcfda2d57922d06c110d9fd77470e"
server_key_version: "encryptionkey@0"
I20260928 23:48:26.437847 12270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:48:26.455982 12270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:48:26.459132 12270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:48:26.459359 12270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:48:26.466938 12270 rpc_server.cc:307] RPC server started. Bound to: 127.21.112.124:39999
I20260928 23:48:26.467190 12338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.112.124:39999 every 8 connection(s)
I20260928 23:48:26.468024 12270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/data/info.pb
I20260928 23:48:26.470973 12339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:48:26.475044 21953 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzqclLI/build/debug/bin/kudu as pid 12270
I20260928 23:48:26.475167 21953 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzqclLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790639014951155-21953-0/minicluster-data/master-2/wal/instance
I20260928 23:48:26.474852 12339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:26.475473 21953 external_mini_cluster.cc:1428] Setting key f6002d6bffd20640dafa3a66638a4ffd
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:48:26.488694 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.476971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.21.112.126:45817)
0928 23:48:26.477108 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.477113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.477568 (+ 455us) server_negotiation.cc:407] Connection header received
0928 23:48:26.478706 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.478709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.478759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.478835 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.479846 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.480348 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.481451 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.481815 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.481898 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.484676 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.484681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.484688 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.484726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.486294 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.486898 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.486901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.486902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.486975 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.487339 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.487342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.487343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.487630 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.487719 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.488332 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.488471 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 23:48:26.489709 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.476889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.126:45817 (local address 127.0.0.1:46912)
0928 23:48:26.477382 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.477427 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.478477 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.478917 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.478922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.479371 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.479646 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.479667 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.480462 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.480466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.481333 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.481341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.481987 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.481992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.482164 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.482837 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.482853 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.484495 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.486401 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.486412 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.486427 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.486795 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.487065 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.487071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.487075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.487247 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.487731 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.487739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.488072 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.488283 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.488656 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":144,"threads_started":1}
I20260928 23:48:26.489709 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.476889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.21.112.124:39999)
0928 23:48:26.477754 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.477766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.477799 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:48:26.478485 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.478508 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.478762 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.478972 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.479698 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.480739 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.481651 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.482153 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.482261 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.485121 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.485129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.485144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.485174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.487197 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.487675 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.487702 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.487704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.487789 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.488057 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.488060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.488061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.488334 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.488512 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.489042 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.489180 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"spinlock_wait_cycles":41344,"thread_start_us":144,"threads_started":1}
I20260928 23:48:26.490962 12339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:26.491834 12188 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } attempt: 3
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.126@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.126@KRBTEST.COM
I20260928 23:48:26.498404 12263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } attempt: 3
I20260928 23:48:26.501231 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.491593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.21.112.125:46679)
0928 23:48:26.491731 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.491735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.491748 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:48:26.491783 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.491785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.491849 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.491931 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.492560 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.493390 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.494426 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.494770 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.494875 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.497443 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.497446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.497449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.497484 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.499346 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.499852 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.499857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.499860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.499951 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.500267 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.500270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.500272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.500489 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.500577 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.500949 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.501076 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:48:26.501700 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.491492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.125:46679 (local address 127.0.0.1:34388)
0928 23:48:26.491617 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.491632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.491698 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.492105 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.492110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.492288 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.492405 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.492414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.493510 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.493514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.494287 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.494297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.494880 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.494884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.495036 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.495521 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.495534 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.497296 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.499474 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.499478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.499480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.499735 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.500065 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.500067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.500068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.500136 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.500655 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.500658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.500759 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.501376 (+ 617us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.501510 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":35}
I20260928 23:48:26.502203 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.492715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.21.112.126:45817)
0928 23:48:26.492822 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.492826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.492898 (+ 72us) server_negotiation.cc:407] Connection header received
0928 23:48:26.493027 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.493033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.493079 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.493154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.493860 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.494325 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.495355 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.495729 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.495786 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.498472 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.498475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.498478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.498505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.500093 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.500577 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.500582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.500584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.500670 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.500983 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.500986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.500987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.501197 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.501480 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.501901 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.502008 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:48:26.502413 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.492222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35620)
0928 23:48:26.492316 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.492332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.492411 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.492679 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.492682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.492856 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.492949 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.492955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.493894 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.493897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.494632 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.494641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.495255 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.495259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.495345 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.495678 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.495694 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.497296 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.500787 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.500790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.500794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.501060 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.501367 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.501372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.501375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.501457 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.501930 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.501933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.502034 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.502115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.502242 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:48:26.502437 12339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:48:26.502991 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.492279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.21.112.124:39999)
0928 23:48:26.492380 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.492384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.492397 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:48:26.492511 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.492525 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.492569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.492640 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.493088 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.493768 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.494743 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.495108 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.495248 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.497441 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.497444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.497446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.497477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.500643 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.501180 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.501183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.501185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.501260 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.501600 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.501602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.501604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.501808 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.502672 (+ 864us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.502741 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.502834 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":42}
I20260928 23:48:26.507035 12188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98: Bootstrap starting.
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.124@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.112.125@KRBTEST.COM
I20260928 23:48:26.510196 12188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:26.510710 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.502854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35634)
0928 23:48:26.503050 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.503084 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.503157 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.503544 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.503546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.503711 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.503793 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.503801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.504605 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.504608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.505091 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.505098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.505562 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.505566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.505634 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.506084 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.506097 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.507409 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.509098 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.509101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.509103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.509332 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.509668 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.509671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.509673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.509764 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.510204 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.510207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.510283 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.510390 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.510520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260928 23:48:26.511098 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.503214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.21.112.124:39999)
0928 23:48:26.503303 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.503307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.503318 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:48:26.503344 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.503346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.503382 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.503448 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.503913 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.504487 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.505202 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.505430 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.505589 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.507547 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.507551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.507553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.507577 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.508982 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.509440 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.509443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.509445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.509572 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.509850 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.509852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.509853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.510035 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.510153 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.510902 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.511001 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:48:26.511250 12188 log.cc:824] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:26.514170 12339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: Bootstrap starting.
Sep 28 23:48:26 dist-test-slave-c3s8 krb5kdc[12107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790639306, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.112.125@KRBTEST.COM for kudu/127.21.112.124@KRBTEST.COM
I20260928 23:48:26.514307 12188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98: No bootstrap required, opened a new log
I20260928 23:48:26.514463 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.503876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.21.112.125:46679)
0928 23:48:26.503981 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.503985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.503998 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:48:26.504041 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.504043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.504081 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.504157 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.504864 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.505514 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.506598 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.506959 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.507074 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.510413 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.510417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.510419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.510448 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.512205 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.512679 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.512683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.512684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.512773 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.513365 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.513369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.513370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.513622 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.513808 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.514190 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.514306 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:48:26.516897 12339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:26.517791 12339 log.cc:824] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:26.518846 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.498814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.112.124:39999 (local address 127.0.0.1:35632)
0928 23:48:26.499074 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 23:48:26.499089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:48:26.499176 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:48:26.499938 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:48:26.499941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:48:26.500109 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:48:26.500219 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.500228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.503557 (+ 3329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.503563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.504344 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:48:26.504357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.512738 (+ 8381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.512744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:48:26.512837 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.513270 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:48:26.513283 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:48:26.514757 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:48:26.517240 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.517243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.517246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.517497 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.517835 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:48:26.517838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:48:26.517840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:48:26.517920 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:48:26.518393 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:48:26.518396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:48:26.518503 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:48:26.518621 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 23:48:26.518737 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260928 23:48:26.519207 21953 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:48:26.519917 12188 raft_consensus.cc:348] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.520192 12188 raft_consensus.cc:729] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda25b336dc84acbbee7b0cc15b50d98, State: Initialized, Role: FOLLOWER
I20260928 23:48:26.520308 12339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: No bootstrap required, opened a new log
I20260928 23:48:26.520503 12154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fda25b336dc84acbbee7b0cc15b50d98"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46914
I20260928 23:48:26.520542 12188 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.520931 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.499323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.21.112.124:39999)
0928 23:48:26.499693 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.499702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.499715 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:48:26.499746 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.499749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.499797 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.499883 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.500343 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.502844 (+ 2501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.512217 (+ 9373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.512594 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.513030 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.515104 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.515107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.515110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.515133 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.517118 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.517651 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.517654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.517656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.517735 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.518074 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.518077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.518079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.518284 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.518385 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.520473 (+ 2088us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.520593 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":150,"threads_started":1}
I20260928 23:48:26.521148 12347 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.521360 12347 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.522061 12188 sys_catalog.cc:564] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:26.522133 12154 raft_consensus.cc:482] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:48:26.522250 12154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:26.523509 12154 raft_consensus.cc:504] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.524150 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: Bootstrap starting.
I20260928 23:48:26.524444 12154 leader_election.cc:288] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [CANDIDATE]: Term 1 election: Requested vote from peers 080c0bdba2f64b20b77bf110de744445 (127.21.112.125:46679), c60fca428b9f46fc862c6be732941039 (127.21.112.124:39999)
W20260928 23:48:26.527779 12358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:26.527860 12358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:26.529400 12263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: Neither blocks nor log segments found. Creating new log.
I20260928 23:48:26.526640 12120 master_runner.cc:186] Error getting master registration for 127.21.112.125:46679: OK, instance_id { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" instance_seqno: 1790639306243593 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:48:26.529903 12120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:48:26.530439 12263 log.cc:824] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: Log is configured to *not* fsync() on all Append() calls
I20260928 23:48:26.530822 12339 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.531432 12339 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60fca428b9f46fc862c6be732941039, State: Initialized, Role: FOLLOWER
I20260928 23:48:26.532120 12339 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.532785 12263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: No bootstrap required, opened a new log
I20260928 23:48:26.532899 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.533087 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.535065 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fda25b336dc84acbbee7b0cc15b50d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c60fca428b9f46fc862c6be732941039"
I20260928 23:48:26.535293 12313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:26.537182 12313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda25b336dc84acbbee7b0cc15b50d98 in term 1.
I20260928 23:48:26.537075 12263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.537287 12263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080c0bdba2f64b20b77bf110de744445, State: Initialized, Role: FOLLOWER
I20260928 23:48:26.537685 12339 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:26.537648 12263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.537827 12137 leader_election.cc:302] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c60fca428b9f46fc862c6be732941039, fda25b336dc84acbbee7b0cc15b50d98; no voters:
I20260928 23:48:26.538206 12347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:48:26.538444 12347 raft_consensus.cc:686] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [term 1 LEADER]: Becoming Leader. State: Replica: fda25b336dc84acbbee7b0cc15b50d98, State: Running, Role: LEADER
I20260928 23:48:26.538790 12347 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } }
I20260928 23:48:26.539125 12263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:48:26.539412 12237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fda25b336dc84acbbee7b0cc15b50d98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "080c0bdba2f64b20b77bf110de744445"
I20260928 23:48:26.539634 12237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:48:26.540583 12347 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda25b336dc84acbbee7b0cc15b50d98. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.540686 12347 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:26.541043 12237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda25b336dc84acbbee7b0cc15b50d98 in term 1.
I20260928 23:48:26.543416 12376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:48:26.544409 12376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:48:26.546635 12270 master_runner.cc:186] Error getting master registration for 127.21.112.125:46679: OK, instance_id { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" instance_seqno: 1790639306243593 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:48:26.549945 12270 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 23:48:26.551882 12394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:26.551975 12394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 23:48:26.552248 12388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:48:26.552352 12388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:48:26.548833 12366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.552591 12366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.558218 12313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [term 1 FOLLOWER]: Refusing update from remote peer fda25b336dc84acbbee7b0cc15b50d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:48:26.558979 12367 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:48:26.560286 12237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [term 1 FOLLOWER]: Refusing update from remote peer fda25b336dc84acbbee7b0cc15b50d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:48:26.560868 12367 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:48:26.563179 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.531277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.21.112.125:46679)
0928 23:48:26.531393 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.531398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.542174 (+ 10776us) server_negotiation.cc:407] Connection header received
0928 23:48:26.542386 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.542389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.542447 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.542504 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.543576 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.544305 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.553816 (+ 9511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.554164 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.554279 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.559448 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.559452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.559455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.559487 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.561262 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.561813 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.561816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.561818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.561900 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.562226 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.562229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.562231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.562452 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.562565 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.562901 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.563033 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:48:26.567557 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda25b336dc84acbbee7b0cc15b50d98. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.567691 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.569952 12366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda25b336dc84acbbee7b0cc15b50d98. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.570093 12366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.574744 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.541453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.21.112.124:39999)
0928 23:48:26.541689 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.541696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.545282 (+ 3586us) server_negotiation.cc:407] Connection header received
0928 23:48:26.545330 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.545333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.545384 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.545469 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.554627 (+ 9158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.555384 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.556560 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.556924 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.557024 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.570321 (+ 13297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.570325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.570328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.570360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.572106 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.572631 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.572635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.572637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.572731 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.573029 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.573033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.573035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.573259 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.573385 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.574179 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.574310 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:48:26.581269 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:26.542301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.21.112.126:45817)
0928 23:48:26.542417 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.542421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.563763 (+ 21342us) server_negotiation.cc:407] Connection header received
0928 23:48:26.566264 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.566268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.566320 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.566375 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.567552 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.568275 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.573636 (+ 5361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:26.574007 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:26.574104 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:26.577683 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:48:26.577687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:48:26.577689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:48:26.577720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:48:26.579329 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.579816 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.579820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.579821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.579905 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:48:26.580236 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:48:26.580242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:48:26.580246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:48:26.580478 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:48:26.580607 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:26.580989 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 23:48:26.581117 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:48:26.582629 12367 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.582770 12367 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:26.582909 12389 mvcc.cc:205] Tried to move back new op lower bound from 7334458599654252544 to 7334458599586459648. Current Snapshot: MvccSnapshot[applied={T|T < 7334458599654252544}]
I20260928 23:48:26.583218 12365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.583357 12365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.585029 12399 mvcc.cc:205] Tried to move back new op lower bound from 7334458599654252544 to 7334458599586459648. Current Snapshot: MvccSnapshot[applied={T|T < 7334458599654252544}]
I20260928 23:48:26.585807 12347 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.585935 12347 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98 [sys.catalog]: This master's current role is: LEADER
I20260928 23:48:26.587100 12366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fda25b336dc84acbbee7b0cc15b50d98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda25b336dc84acbbee7b0cc15b50d98" member_type: VOTER last_known_addr { host: "127.21.112.126" port: 45817 } } peers { permanent_uuid: "080c0bdba2f64b20b77bf110de744445" member_type: VOTER last_known_addr { host: "127.21.112.125" port: 46679 } } peers { permanent_uuid: "c60fca428b9f46fc862c6be732941039" member_type: VOTER last_known_addr { host: "127.21.112.124" port: 39999 } } }
I20260928 23:48:26.587236 12366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:48:26.587776 12398 mvcc.cc:205] Tried to move back new op lower bound from 7334458599654252544 to 7334458599586459648. Current Snapshot: MvccSnapshot[applied={T|T < 7334458599654252544}]
I20260928 23:48:26.594482 12376 catalog_manager.cc:1347] Generated new cluster ID: 9f089dbbcd51408194213bb15f050912
I20260928 23:48:26.594566 12376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:48:26.641765 12376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:48:26.643227 12376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:48:26.666946 12376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fda25b336dc84acbbee7b0cc15b50d98: Generated new TSK 0
I20260928 23:48:26.667865 12376 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:48:26.701596 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.698436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.21.112.125:46679)
0928 23:48:26.698578 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.698583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.700369 (+ 1786us) server_negotiation.cc:407] Connection header received
0928 23:48:26.700639 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.700643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.700695 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.700748 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.701410 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34404: BlockingRecv error: recv got EOF from 127.0.0.1:34404 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 23:48:26.703002 12344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.699148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.21.112.124:39999)
0928 23:48:26.699256 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.699260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.702103 (+ 2843us) server_negotiation.cc:407] Connection header received
0928 23:48:26.702276 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.702279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.702328 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.702386 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.702888 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35654: BlockingRecv error: recv got EOF from 127.0.0.1:35654 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 23:48:26.704564 12192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.699784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.21.112.126:45817)
0928 23:48:26.699892 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.699896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.703395 (+ 3499us) server_negotiation.cc:407] Connection header received
0928 23:48:26.703784 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.703788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.703842 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.703894 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.704430 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46932: BlockingRecv error: recv got EOF from 127.0.0.1:46932 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 23:48:26.709059 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.707540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.21.112.125:46679)
0928 23:48:26.707643 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.707646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.708099 (+ 453us) server_negotiation.cc:407] Connection header received
0928 23:48:26.708309 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.708312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.708383 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.708436 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.708957 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34406: BlockingRecv error: recv got EOF from 127.0.0.1:34406 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 23:48:26.715010 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.713393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.21.112.125:46679)
0928 23:48:26.713508 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.713512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.714062 (+ 550us) server_negotiation.cc:407] Connection header received
0928 23:48:26.714259 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.714263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.714315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.714391 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.714907 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34414: BlockingRecv error: recv got EOF from 127.0.0.1:34414 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 23:48:26.721252 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.719473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.21.112.125:46679)
0928 23:48:26.719577 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.719582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.720125 (+ 543us) server_negotiation.cc:407] Connection header received
0928 23:48:26.720314 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.720317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.720404 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.720478 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.721142 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34416: BlockingRecv error: recv got EOF from 127.0.0.1:34416 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 23:48:26.729761 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.727924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.21.112.125:46679)
0928 23:48:26.728096 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.728101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.728640 (+ 539us) server_negotiation.cc:407] Connection header received
0928 23:48:26.728853 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.728857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.728921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.728978 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.729641 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34418: BlockingRecv error: recv got EOF from 127.0.0.1:34418 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 23:48:26.742149 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.740616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.21.112.125:46679)
0928 23:48:26.740749 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.740754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.741046 (+ 292us) server_negotiation.cc:407] Connection header received
0928 23:48:26.741273 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.741277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.741338 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.741435 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.742012 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34420: BlockingRecv error: recv got EOF from 127.0.0.1:34420 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 23:48:26.761745 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.760397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.21.112.125:46679)
0928 23:48:26.760531 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.760536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.760789 (+ 253us) server_negotiation.cc:407] Connection header received
0928 23:48:26.760967 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.760977 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.761030 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.761122 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.761620 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34426: BlockingRecv error: recv got EOF from 127.0.0.1:34426 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 23:48:26.797031 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.795884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.21.112.125:46679)
0928 23:48:26.796038 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.796042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.796186 (+ 144us) server_negotiation.cc:407] Connection header received
0928 23:48:26.796306 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.796308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.796356 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.796440 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.796885 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34436: BlockingRecv error: recv got EOF from 127.0.0.1:34436 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 23:48:26.864445 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.863344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.21.112.125:46679)
0928 23:48:26.863506 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.863510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.863654 (+ 144us) server_negotiation.cc:407] Connection header received
0928 23:48:26.863773 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.863776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.863825 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.863909 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.864346 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34448: BlockingRecv error: recv got EOF from 127.0.0.1:34448 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 23:48:26.996042 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:26.994813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.21.112.125:46679)
0928 23:48:26.995009 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:26.995013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:26.995219 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:48:26.995305 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:26.995308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:26.995360 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:26.995463 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:26.995889 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34454: BlockingRecv error: recv got EOF from 127.0.0.1:34454 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260928 23:48:27.255614 12269 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:48:27.254295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.21.112.125:46679)
0928 23:48:27.254450 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:27.254454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:27.254651 (+ 197us) server_negotiation.cc:407] Connection header received
0928 23:48:27.254833 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:27.254841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:27.254893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:27.254988 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:48:27.255469 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34458: BlockingRecv error: recv got EOF from 127.0.0.1:34458 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 23:48:27.553835 12388 catalog_manager.cc:1259] Loaded cluster ID: 9f089dbbcd51408194213bb15f050912
I20260928 23:48:27.553846 12394 catalog_manager.cc:1259] Loaded cluster ID: 9f089dbbcd51408194213bb15f050912
I20260928 23:48:27.553922 12394 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: loading cluster ID for follower catalog manager: success
I20260928 23:48:27.553922 12388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: loading cluster ID for follower catalog manager: success
I20260928 23:48:27.556571 12388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: acquiring CA information for follower catalog manager: success
I20260928 23:48:27.556571 12394 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: acquiring CA information for follower catalog manager: success
I20260928 23:48:27.558421 12388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c60fca428b9f46fc862c6be732941039: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:48:27.558439 12394 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 080c0bdba2f64b20b77bf110de744445: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:48:27.774072 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:27.769956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.21.112.125:46679)
0928 23:48:27.770214 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:27.770217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:27.770233 (+ 16us) server_negotiation.cc:407] Connection header received
0928 23:48:27.770352 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:27.770354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:27.770479 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:27.770586 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:27.770933 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.771466 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.772376 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.772641 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.772729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:27.773050 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:27.773519 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:27.773702 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:27.773798 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:48:27.773862 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260928 23:48:27.782456 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:27.777408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.21.112.124:39999)
0928 23:48:27.777685 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:27.777689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:27.777706 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:48:27.777918 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:27.777922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:27.778090 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:27.778254 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:27.778488 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.779215 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.780209 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.780606 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.780720 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:27.781134 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:27.781802 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:27.781965 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:27.782172 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 23:48:27.782257 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
I20260928 23:48:27.791239 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:27.786694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.21.112.124:39999)
0928 23:48:27.786825 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:27.786829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:27.787328 (+ 499us) server_negotiation.cc:407] Connection header received
0928 23:48:27.787457 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:27.787460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:27.787520 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:27.787607 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:27.787830 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.788643 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.789876 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.790185 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.790285 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:27.790747 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:27.790830 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:27.790876 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:27.791041 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 23:48:27.791093 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:48:27.791266 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:27.785865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.21.112.126:45817)
0928 23:48:27.786153 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:27.786157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:27.786551 (+ 394us) server_negotiation.cc:407] Connection header received
0928 23:48:27.786748 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:27.786752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:27.786920 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:27.787059 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:27.787309 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.788152 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.789398 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.789816 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.789894 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:27.790338 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:27.790727 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:27.790810 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:27.790975 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 23:48:27.791065 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260928 23:48:27.791460 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:48:27.786069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.21.112.125:46679)
0928 23:48:27.786202 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:48:27.786207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:48:27.786889 (+ 682us) server_negotiation.cc:407] Connection header received
0928 23:48:27.787128 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:48:27.787131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:48:27.787207 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:48:27.787300 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:48:27.787661 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.788375 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.789508 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:48:27.789938 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:48:27.790045 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:48:27.790808 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:48:27.790902 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:48:27.790970 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:48:27.791158 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:48:27.791262 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:48:27.795014 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 12120
I20260928 23:48:27.802251 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 12195
I20260928 23:48:27.808306 21953 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzqclLI/build/debug/bin/kudu with pid 12270
2026-09-28T23:48:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3166 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6301 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292847 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST